Extended tests against shareware
Total of 3081 unique (shareware) Wise installers have been found, this is good for testing. With these extended test scripts, written in Python; make it easy to see what installers broke or fixed after code changes. SEE the new READMEs for more info.
This commit is contained in:
CYBERDEV
parent
5121c5581d
commit
f921650601
16 changed files with 18630 additions and 1 deletions
|
|
@ -9,7 +9,7 @@ end_of_line = lf
|
|||
insert_final_newline = true
|
||||
|
||||
# 2 space indentation
|
||||
[*.{c,h,hexpat}]
|
||||
[*.{c,h,py,hexpat}]
|
||||
charset = utf-8
|
||||
indent_style = space
|
||||
indent_size = 2
|
||||
|
|
|
|||
1
.gitignore
vendored
1
.gitignore
vendored
|
|
@ -4,3 +4,4 @@ src/*.o
|
|||
|
||||
# Test cache
|
||||
tests/cache/
|
||||
tests/extended/cache/
|
||||
|
|
|
|||
236
tests/extended/README.md
Normal file
236
tests/extended/README.md
Normal file
|
|
@ -0,0 +1,236 @@
|
|||
# Extended tests
|
||||
|
||||
> NOTE: This is for developers
|
||||
|
||||
A large number (current 3081) of unique shareware Wise installers have
|
||||
been found. It is `70.19 GiB` in size, so make sure you have that free space.
|
||||
|
||||
These tests won't extract anything to disk, it uses `--list`, `--raw-verify`,
|
||||
and `--verify` for the dev version of REWise and do a `--verify` test with
|
||||
REWise v0.2.0 for comparison. Along with that it tries to analyze errors and
|
||||
comment them.
|
||||
|
||||
These scripts require Python 3, so make sure you have that installed.
|
||||
|
||||
> TIP: Use `csvlens`, `libreoffice` or whatever to view the .csv files.
|
||||
|
||||
## How to
|
||||
|
||||
### Initial setup
|
||||
|
||||
1. If you don't have a REWise v0.2.0 binary then download and compile
|
||||
[REWise v0.2.0](https://codeberg.org/CYBERDEV/REWise/archive/v0.2.0.tar.gz).
|
||||
2. Compile the REWise dev version with `make debug`, it should be
|
||||
in `../../` (from this README.md file, by default).
|
||||
3. Copy `config.py.template` to `config.py` and edit `REWISE_02_PATH`
|
||||
so it points to your REWise v0.2.0 binary. When you already have
|
||||
downloaded the shareware installers somewhere else then default
|
||||
you want to edit `DL_PATH`.
|
||||
4. Download the installers; make sure you have enough free space and take in
|
||||
mind that this will take a while:
|
||||
|
||||
```
|
||||
python test.py download sources/shareware.csv
|
||||
```
|
||||
|
||||
See the `cache/dl.log` file for download/file errors.
|
||||
|
||||
5. Run the initial test; gather info about the installers, run tests against
|
||||
REWise v0.2.0 and the REWise dev version. The input will be a source .csv
|
||||
and the output will be a result .csv file that will be our base. Later
|
||||
tests will skip the REWise v0.2.0 tests because they will be the
|
||||
same anyway.
|
||||
|
||||
```
|
||||
python test.py init sources/shareware.csv ./results.csv
|
||||
```
|
||||
|
||||
|
||||
### Testing local changes
|
||||
|
||||
So you made some code changes and recompiled REWise with `make debug`
|
||||
(`rewise` bin should be in `../../`), now it's time to see if it fixed
|
||||
or broke any installer.
|
||||
|
||||
First thing is to re-run the tests against the freshly compiled REWise
|
||||
bin:
|
||||
|
||||
```
|
||||
python test.py create-diff ./results.csv ./new_results.csv
|
||||
```
|
||||
|
||||
To print the installers that it fixed/broke:
|
||||
|
||||
```
|
||||
python test.py print-diff ./results.csv ./new_results.csv
|
||||
|
||||
Fixed (0)
|
||||
--------------------------------
|
||||
|
||||
Broke (2)
|
||||
--------------------------------
|
||||
c7aa23ed03082d671a1e2bf776361d685b51f1acfa9525a636f15e25d681084e setup.exe
|
||||
ff9c38732523f39432479a1f5625a7bf089fab036cc5c7d89f288df1bf56e306 AdminPro_DEMO.exe
|
||||
```
|
||||
|
||||
To print stats of the new results:
|
||||
|
||||
```
|
||||
python test.py print-stats ./new_results.csv
|
||||
|
||||
| DATE | OK | ERR | LOK | LERR | ROK | RERR | v02 | v02E | VERSION
|
||||
| :------------------ | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | :------
|
||||
| NE | 96 | 6 | 100 | 2 | 101 | 1 | 0 | 102 | Unknown
|
||||
| 1998-10-02 22:36:51 | 78 | 4 | 78 | 4 | 82 | 0 | 58 | 24 | Unknown
|
||||
| 1998-10-23 23:23:16 | 2 | 0 | 2 | 0 | 2 | 0 | 2 | 0 | Unknown
|
||||
| 1998-11-09 21:17:09 | 176 | 22 | 179 | 19 | 198 | 0 | 132 | 66 | InstallMaster 7
|
||||
| 1998-12-03 23:11:32 | 43 | 0 | 43 | 0 | 43 | 0 | 28 | 15 | Unknown
|
||||
| 1999-02-05 23:07:52 | 214 | 19 | 233 | 0 | 232 | 1 | 132 | 101 | Unknown
|
||||
| 1999-04-05 18:07:26 | 4 | 0 | 4 | 0 | 4 | 0 | 4 | 0 | Unknown
|
||||
| 1999-05-21 22:48:48 | 406 | 33 | 420 | 19 | 432 | 7 | 267 | 172 | Unknown
|
||||
| 1999-08-17 17:25:48 | 238 | 24 | 250 | 12 | 254 | 8 | 143 | 119 | Unknown
|
||||
| 2000-04-25 16:37:12 | 894 | 49 | 914 | 29 | 937 | 6 | 514 | 429 | InstallMaster 8
|
||||
| 2001-08-13 19:13:38 | 319 | 8 | 325 | 2 | 323 | 4 | 222 | 105 | Unknown
|
||||
| 2001-10-25 21:47:11 | 418 | 24 | 433 | 9 | 429 | 13 | 246 | 196 | Installation System 9
|
||||
| 2031-05-20 22:54:01 | 4 | 0 | 4 | 0 | 4 | 0 | 4 | 0 | Unknown
|
||||
| ------------------- | ---- | ---- | ---- | ---- | ---- | ---- | ---- | ---- | -------
|
||||
| Total: 3081 | 2892 | 189 | 2985 | 96 | 3041 | 40 | 1752 | 1329 |
|
||||
|
||||
PK Total : 438
|
||||
PK OK : 432
|
||||
PK ERROR : 6
|
||||
PK Raw OK : 438
|
||||
PK Raw ERROR: 0
|
||||
|
||||
Comments:
|
||||
80 Without comment
|
||||
23 Negative script deflate offset
|
||||
21 Possible weird stuff at WiseScriptHeader.unknown_22 '9190A09A878B'.
|
||||
14 Possible weird stuff at WiseScriptHeader.unknown_22 '958A928F8C8B'.
|
||||
10 Possible weird stuff at WiseScriptHeader.unknown_22 'B890BE889E86'.
|
||||
6 Possible weird stuff at WiseScriptHeader.unknown_22 'B6BF88BF96BF'.
|
||||
6 Possible weird stuff at WiseScriptHeader.unknown_22 'A8B0ADB3BBB6'.
|
||||
5 'Initializing Wise Installation Wizard...' found in the PE b'.data\x00\x00\x00' section.
|
||||
3 Possible weird stuff at WiseScriptHeader.unknown_22 'BCADA6CEA5CD'.
|
||||
3 Possible weird stuff at WiseScriptHeader.unknown_22 'C8CACBC8C6CC'.
|
||||
2 Possible weird stuff at WiseScriptHeader.unknown_22 '99939E9D919E'.
|
||||
2 0x18 mismatch, 0x18 00 08 ..
|
||||
2 Possible weird stuff at WiseScriptHeader.unknown_22 'DB8F9E8C8C88'.
|
||||
2 Possible weird stuff at WiseScriptHeader.unknown_22 'ADBF90BF93BF'.
|
||||
1 Possible weird stuff at WiseScriptHeader.unknown_22 '8588C9C6C88B'.
|
||||
1 Possible weird stuff at WiseScriptHeader.unknown_22 'AB969C949A8B'.
|
||||
1 0x18 mismatch, there is OP 0x00 right after 0x18
|
||||
1 Script deflate offset is absurd high
|
||||
1 Possible weird stuff at WiseScriptHeader.unknown_22 'CC918B90929D'.
|
||||
1 Possible weird stuff at WiseScriptHeader.unknown_22 'B4AAACB6B5AC'.
|
||||
1 Possible weird stuff at WiseScriptHeader.unknown_22 '9D8D9E919B96'.
|
||||
1 Possible weird stuff at WiseScriptHeader.unknown_22 '9C909B9A989A'.
|
||||
1 Possible weird stuff at WiseScriptHeader.unknown_22 '8C929E9393D1'.
|
||||
1 Overlay header unknown2 != 0x8000 but 0x0000.
|
||||
|
||||
Verify errors:
|
||||
75 WS_ERROR_VALUE ERROR: parseWiseScriptHeader failed to read languageTexts.
|
||||
46 WS_ERROR_INFLATE ERROR: Inflated size is larger then expected
|
||||
25 WS_ERROR_SEEK ERROR: Failed seek to file offset
|
||||
9 ERROR_VALUE ERROR: Advertised WiseScript.bin size looks insane.
|
||||
7 WS_ERROR_INFLATE ERROR: zlib: invalid code lengths set
|
||||
4 WS_ERROR_INFLATE ERROR: zlib: invalid distance too far back
|
||||
4 WS_ERROR_INFLATE ERROR: zlib: invalid stored block lengths
|
||||
3 WS_ERROR_PK_HEADER ERROR: PK signature unknown
|
||||
3 WS_ERROR_VALUE ERROR: wiseScriptParsePath expected '%' at path start
|
||||
2 WS_ERROR_VALUE ERROR: parseWiseScript unknown OP: 3A
|
||||
2 WS_ERROR_INFLATE ERROR: zlib: invalid block type
|
||||
2 ERROR_VALUE ERROR: Advertised .dib size looks insane.
|
||||
1 WS_ERROR_BRANCHING ERROR: OP 0x23 LANG but not in LANG branch
|
||||
1 WS_ERROR_INFLATE ERROR: zlib: invalid literal/length code
|
||||
1 WS_ERROR_VALUE ERROR: This is a multi language installer but not all languages have their name set
|
||||
1 WS_ERROR_VALUE ERROR: parseWiseScript unknown OP: 49
|
||||
1 WS_ERROR_INFLATE ERROR: zlib: invalid distance code
|
||||
1 WS_ERROR_BRANCHING ERROR: OP 0x23 LANG but currently not in a branch
|
||||
1 ERROR_INFLATE ERROR: zlib: invalid stored block lengths
|
||||
```
|
||||
|
||||
Further inspection of the results file can be done using something that can
|
||||
view .csv files.
|
||||
|
||||
|
||||
### Write debug scripts
|
||||
|
||||
I've written a lot of small scripts for debugging specific stuff, like printing
|
||||
a list of all PE build dates and the overlay offset(s) observed. Such things
|
||||
are very specific and I didn't find it worth including, but here is a example:
|
||||
|
||||
```python
|
||||
import os
|
||||
from installers import installerStatsCsvIt
|
||||
|
||||
found = {}
|
||||
for installer in installerStatsCsvIt("new_results.csv"):
|
||||
if installer.isNe:
|
||||
continue
|
||||
|
||||
fdate = installer.formattedDate()
|
||||
if fdate not in found:
|
||||
found.update({fdate: []});
|
||||
|
||||
hexoverlay = hex(installer.overlay)
|
||||
if hexoverlay not in found[fdate]:
|
||||
found[fdate].append(hexoverlay)
|
||||
|
||||
for pebuild, offsets in sorted(found.items()):
|
||||
print(pebuild, offsets)
|
||||
```
|
||||
|
||||
## Files
|
||||
|
||||
### test.py
|
||||
|
||||
```help
|
||||
usage: test.py [-h] {download,init,update,create-diff,filter-invalid,update-comments,print-diff,print-stats,result-to-source} ...
|
||||
|
||||
options:
|
||||
-h, --help show this help message and exit
|
||||
|
||||
COMMANDS:
|
||||
Note: use -h after a command to get more help on that command
|
||||
|
||||
{download,init,update,create-diff,filter-invalid,update-comments,print-diff,print-stats,result-to-source}
|
||||
download Download installers from the given SOURCE file when needed
|
||||
init Run initial test, the output will be written to the given RESULT file. The initial test does include testing against REWise v0.2
|
||||
update Add new installers from given SOURCE and re-run tests for all installers, excluding the REWise v0.2 tests. To only add new and run tests for new installers give the '--only-new' argument.
|
||||
create-diff Re-run tests of first given RESULT, ouput to second given RESULT file.
|
||||
filter-invalid Filter out results that appear to be invalid files / not Wise installers.
|
||||
update-comments Update comments for given RESULT file.
|
||||
print-diff Compare two RESULT files and print the installers that it fixed/broke.
|
||||
print-stats Print some stats for the given RESULT file.
|
||||
result-to-source Convert RESULT file into SOURCE file.
|
||||
```
|
||||
|
||||
#### `module` analyze.py
|
||||
|
||||
Analyze errors and add comments.
|
||||
|
||||
#### `module` dl.py
|
||||
|
||||
For downloading files.
|
||||
|
||||
#### `module` sources.py
|
||||
|
||||
Handle reading source .csv files.
|
||||
|
||||
The columns:
|
||||
|
||||
1. B3 (Blake 3)
|
||||
2. MD5
|
||||
3. Filesize in bytes
|
||||
4. Download URL
|
||||
|
||||
#### `module` installers.py
|
||||
|
||||
Handle reading/writing result .csv files.
|
||||
|
||||
#### `module` rewise.py
|
||||
|
||||
This module handles testing and output parsing of REWise current dev and
|
||||
REWise v0.2.0.
|
||||
|
||||
127
tests/extended/analyze.py
Normal file
127
tests/extended/analyze.py
Normal file
|
|
@ -0,0 +1,127 @@
|
|||
# Analyze errors
|
||||
import os
|
||||
import pefile
|
||||
import zipfile
|
||||
|
||||
|
||||
def addCustomComments(installer):
|
||||
if installer.md5 == "32aad22244ee0c8e7dcd3f86dea9e347": # NPFG10.EXE
|
||||
installer.addComment("This one has '../' in one of it's filenames, it is resolved by escaping the '../' with '\\.\\./'")
|
||||
|
||||
elif installer.md5 == "a91a9034e99da6917ed6ea4ae1017474": # b65full.exe
|
||||
installer.addComment("Corrupt, missing data, see "
|
||||
"c3d819097d06d110f1b01111a89b095c for valid "
|
||||
"data.")
|
||||
|
||||
elif installer.md5 in ["70a6bbf41b7e9de48e711dbc26eac88b", # dap43.exe
|
||||
"f463a1d47ac919160fc450d322c6c63e", # dap4.exe
|
||||
"7d093e7e91b17b0b6ca427c91202c839"]: # dap3908.exe
|
||||
installer.addComment("0x18 mismatch, there is OP 0x00 right after 0x18")
|
||||
|
||||
elif installer.md5 in ["5516bfdc0346f475300f83ebc6c76547", # lw3drv.exe
|
||||
"3be4f8d1fc6ede81d954b6914530dd2b", # lw3w2k.exe
|
||||
"1588bc310e9a71389f1069af159203ee", # lw3wme.exe
|
||||
"073c6e77547bcda5ce7b5ebe96f3eb01"]:# SBL51.EXE
|
||||
installer.addComment("0x18 mismatch")
|
||||
|
||||
elif installer.md5 in ["7c763e4af5aca8cc2fabcc846b788692", # DSiegeUpdate1_09_2-1_1_1460.exe
|
||||
"acccf3d47143f0cf2b6b893765586eb6", # Dsiegeupdate1.0-1.11.1462_english.exe
|
||||
"3465e2b1eec01cc26b898c4ef3ef090e"]: # dsupdate1.1-1.11.1462_eng.exe
|
||||
installer.addComment("0x18 mismatch, 0x18 00 08 ..")
|
||||
|
||||
#elif installer.md5 in ["2b4e75b6c5b1ae0fd9243d0cf8db45e5" # psp700ev.exe
|
||||
# "215506a69c9677e9b01bd5c43314ee5f"]: # RD2_onlinedemo_03_10.exe
|
||||
# installer.addComment("False positive, this is a InstallShield installer")
|
||||
|
||||
elif installer.md5 in ["dbbf3189c879ad7673d5971e457effeb", # DPPopUpStopper.exe
|
||||
"ae59de2259f3a109a6d66eb037da2335", # ICQ2000B.EXE
|
||||
"60aef2e657c05cfff2bfbc75d6385f3d"]: # DontPanc.exe
|
||||
installer.addComment("This installer reads the overlay header 3 bytes to early")
|
||||
|
||||
#elif installer.md5 in ["a44f7489825a35b8ea49857d2d832ce6", # thmpls450_s.exe
|
||||
# "cf8e337b9c331cdfbe490a023b184a9f", # icq99b.exe
|
||||
# "058f7c9124cd32463e8dfebe4eb115c7"]: # icq2000a.exe
|
||||
# installer.addComment("FIXME Weird extra chars in unknown_22")
|
||||
|
||||
elif installer.md5 in ["3379f7b191a5f9ad9dee9d49acf99e23", # upgrade.exe
|
||||
"3319738353934531293f8b7e1c103a85"]: # upgrade.exe
|
||||
installer.addComment("Interesting one, this has no 0x00 files, only 0x12, online installer?")
|
||||
|
||||
elif installer.md5 == "28b29549010d0980d1cd1d0ad86f8dbe": # battlefield2demo.exe
|
||||
installer.addComment("False positive, InstallShield Wizard?")
|
||||
|
||||
elif installer.md5 in ["32a6454ef70526f8d80be1069749bfde", # sumofree.exe (overlay: 0x00012200)
|
||||
"3dd7c98df5e6a5e528d446f4ebce46c8", # RD210.EXE (overlay: 0x00012C00)
|
||||
"3c50c1616008c582c7ea3c34fef90d91"]: # empireearth2_mpdemo_en.exe (overlay: 0x0001F800)
|
||||
installer.addComment("False positive? Raw files appended at overlay with probably a Wise installer")
|
||||
|
||||
elif installer.md5 in ["7c420b5be50635f3a2f73cf8e5c490c5", # spf.exe
|
||||
"99c37f56b1a030ed4b04e0b2cb611051", # spf.exe
|
||||
"c087ee0bbd1baf2a6e04151b76fc1216"]: # spf.exe
|
||||
installer.addComment("False positive")
|
||||
|
||||
|
||||
def addPeComments(installer):
|
||||
pe = None
|
||||
try:
|
||||
pe = pefile.PE(installer.realpath, fast_load=True)
|
||||
except pefile.PEFormatError as err:
|
||||
installer.addComment(f"This not a PE file: {err}")
|
||||
return True
|
||||
|
||||
# find the magic string in any PE section and note the offset after read
|
||||
magicoffset = 0
|
||||
sectionName = ""
|
||||
with open(installer.realpath, "rb") as fp:
|
||||
for section in pe.sections:
|
||||
if section.Name[:5] == b'.WISE':
|
||||
installer.addComment("PE .WISE section found.")
|
||||
|
||||
# Read each section into mem and search for the magic string
|
||||
for section in pe.sections:
|
||||
fp.seek(section.VirtualAddress, 0)
|
||||
sectiondata = fp.read(section.SizeOfRawData)
|
||||
if b"Initializing Wise Installation Wizard..." in sectiondata:
|
||||
magicoffset = fp.tell()
|
||||
sectionName = section.Name
|
||||
break
|
||||
|
||||
if magicoffset != 0:
|
||||
installer.addComment(f"'Initializing Wise Installation Wizard...' found in the PE {sectionName} section.")
|
||||
pe.close()
|
||||
return True
|
||||
|
||||
overlay = pe.get_overlay_data_start_offset()
|
||||
if overlay:
|
||||
filesize = os.path.getsize(installer.realpath)
|
||||
delta = filesize - overlay
|
||||
if delta < 4096:
|
||||
installer.addComment(f"Very small overlay size of {delta} bytes")
|
||||
pe.close()
|
||||
return True
|
||||
else:
|
||||
installer.addComment(f"No overlay data")
|
||||
pe.close()
|
||||
return True
|
||||
|
||||
with open(installer.realpath, "rb") as fp:
|
||||
fp.seek(overlay, 0)
|
||||
if fp.read(13) == b'InstallShield':
|
||||
installer.addComment("False positive, 'InstallShield' found at overlay.")
|
||||
pe.close()
|
||||
return True
|
||||
|
||||
pe.close()
|
||||
return False
|
||||
|
||||
|
||||
def falsePositivePkCheck(installer):
|
||||
try:
|
||||
zfp = zipfile.ZipFile(installer.realpath, 'r')
|
||||
zfp.close()
|
||||
except zipfile.BadZipFile:
|
||||
installer.isPk = False
|
||||
installer.addComment("False positive PK, check the overlay header")
|
||||
return True
|
||||
return False
|
||||
|
||||
35
tests/extended/config.py.template
Normal file
35
tests/extended/config.py.template
Normal file
|
|
@ -0,0 +1,35 @@
|
|||
import os
|
||||
# Do not edit this, it is this script's local directory
|
||||
BASE_PATH = os.path.abspath(os.path.dirname(__file__))
|
||||
|
||||
# You probably don't want to change this but when you do point it to
|
||||
# where the source .csv files are.
|
||||
SOURCE_PATH = os.path.abspath(os.path.join(BASE_PATH, "sources/"))
|
||||
|
||||
# When you already have the installers downloaded somewhere point it
|
||||
# there, else it will download them (when not found) to the set location.
|
||||
#CACHE_PATH = os.path.abspath(os.path.join(BASE_PATH, "../cache/dl/"))
|
||||
CACHE_PATH = os.path.abspath(os.path.join(BASE_PATH, "cache/"))
|
||||
|
||||
# Download path
|
||||
DL_PATH = os.path.abspath(os.path.join(CACHE_PATH, "dl/"))
|
||||
|
||||
# Download error log path
|
||||
DL_LOG = os.path.join(CACHE_PATH, "dl.log")
|
||||
|
||||
# Filepath to the 'rewise' bin
|
||||
REWISE_DEV_PATH = os.path.abspath(os.path.join(BASE_PATH, "../../rewise"))
|
||||
|
||||
# DO SET THIS!
|
||||
# Filepath to the 'rewise' v0.2 bin
|
||||
REWISE_02_PATH = os.path.abspath(os.path.join(CACHE_PATH, "rewise_02"))
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
print("BASE_PATH", BASE_PATH)
|
||||
print("CACHE_PATH", CACHE_PATH)
|
||||
print("DL_PATH", DL_PATH)
|
||||
print("DL_LOG", DL_LOG)
|
||||
print("SOURCE_PATH", SOURCE_PATH)
|
||||
print("REWISE_DEV_PATH", REWISE_DEV_PATH)
|
||||
print("REWISE_02_PATH", REWISE_02_PATH)
|
||||
30
tests/extended/dl.py
Normal file
30
tests/extended/dl.py
Normal file
|
|
@ -0,0 +1,30 @@
|
|||
import os
|
||||
import urllib
|
||||
import urllib.error
|
||||
import urllib.request
|
||||
from urllib.parse import urlparse, quote, unquote
|
||||
from http.client import RemoteDisconnected
|
||||
|
||||
|
||||
def encodeUrl(url):
|
||||
parsed = urlparse(url)
|
||||
parsed = parsed._replace(path=quote(unquote(parsed.path)),
|
||||
query=quote(unquote(parsed.query)))
|
||||
return parsed.geturl()
|
||||
|
||||
|
||||
def downloadFile(url, outputFile):
|
||||
url = encodeUrl(url)
|
||||
|
||||
# Create needed dirs
|
||||
outputDir = os.path.dirname(outputFile)
|
||||
if not os.path.isdir(outputDir):
|
||||
os.makedirs(outputDir)
|
||||
|
||||
# Download ..
|
||||
try:
|
||||
urllib.request.urlretrieve(url, outputFile)
|
||||
except (urllib.error.ContentTooShortError, urllib.error.URLError, RemoteDisconnected) as e:
|
||||
print(e, url)
|
||||
return str(e)
|
||||
return None
|
||||
238
tests/extended/installers.py
Normal file
238
tests/extended/installers.py
Normal file
|
|
@ -0,0 +1,238 @@
|
|||
import os
|
||||
import csv
|
||||
import datetime
|
||||
|
||||
from config import BASE_PATH, DL_PATH, SOURCE_PATH
|
||||
import rewise
|
||||
import sources
|
||||
import analyze
|
||||
|
||||
""" -- InstallerStats
|
||||
|
||||
Generate or read build stats
|
||||
"""
|
||||
|
||||
class InstallerStats:
|
||||
__DATA = {
|
||||
"title" : "",
|
||||
"filename" : "",
|
||||
"b3" : "",
|
||||
"md5" : "",
|
||||
"peBuild" : 0,
|
||||
"scriptBuild" : 0,
|
||||
"overlay" : 0,
|
||||
"scriptDeflateOffset": 0,
|
||||
"isNe" : 0, # bool
|
||||
"isPk" : 0, # bool
|
||||
"langCount" : 0,
|
||||
"devStatus" : rewise.getRewiseStatusString(rewise.REWISE_STATUS_UNSET),
|
||||
"devError" : "",
|
||||
"devListStatus": rewise.getRewiseStatusString(rewise.REWISE_STATUS_UNSET),
|
||||
"devRawStatus": rewise.getRewiseStatusString(rewise.REWISE_STATUS_UNSET),
|
||||
"devRawError" : "",
|
||||
"rew02Status" : rewise.getRewiseStatusString(rewise.REWISE_STATUS_UNSET),
|
||||
"rew02Error" : "",
|
||||
"comment" : ""
|
||||
#"dlUrl" : "",
|
||||
#"filepath" : ""
|
||||
}
|
||||
|
||||
__FANCY_NAMES = [
|
||||
"Title",
|
||||
"Filename",
|
||||
"B3",
|
||||
"MD5",
|
||||
"PE Build",
|
||||
"Script Build",
|
||||
"Overlay offset",
|
||||
"Script deflate file offset",
|
||||
"NE",
|
||||
"PK",
|
||||
"Languages",
|
||||
"Dev status",
|
||||
"Dev error",
|
||||
"Dev list status",
|
||||
"Dev raw status",
|
||||
"Dev raw error",
|
||||
"0.2 status",
|
||||
"0.2 error",
|
||||
"Comments"
|
||||
#"Download URL",
|
||||
#"Local filepath"
|
||||
]
|
||||
|
||||
def __init__(self, source=None, rowData=None):
|
||||
if not source and not rowData:
|
||||
raise Exception("InstallerStats: give either source or rowData")
|
||||
|
||||
self.__data = dict(InstallerStats.__DATA)
|
||||
self.__keys = list(self.__data.keys())
|
||||
self.__keyCount = len(self.__keys)
|
||||
self.__columns = [col for col in self.__data.keys()]
|
||||
|
||||
# Load, rowData should be a list
|
||||
if rowData:
|
||||
colIndex = 0
|
||||
for col in rowData:
|
||||
key = self.__keys[colIndex]
|
||||
dtype = type(InstallerStats.__DATA[key])
|
||||
self.__data[key] = dtype(col)
|
||||
colIndex += 1
|
||||
self.filepath = os.path.join(DL_PATH, self.b3)
|
||||
# Gather stats
|
||||
else:
|
||||
self.__data["b3"] = source.b3
|
||||
self.__data["md5"] = source.md5
|
||||
self.__data["dlUrl"] = source.dlUrl
|
||||
self.__data["filename"] = source.getFilename()
|
||||
self.filepath = source.filepath
|
||||
|
||||
# tmp
|
||||
self.overlayHeaderUnk2 = ""
|
||||
self.wsHeaderUnk22 = ""
|
||||
self.realpath = os.path.join(DL_PATH, self.filepath)
|
||||
|
||||
def __getattr__(cls, key):
|
||||
return cls.__data[key]
|
||||
|
||||
def __str__(self):
|
||||
return str(self.__data)
|
||||
|
||||
# for sorting on b3
|
||||
def __lt__(self, other):
|
||||
return self.b3 < other.b3
|
||||
|
||||
def addComment(self, comment):
|
||||
if self.__data["comment"]:
|
||||
self.__data["comment"] += "\n---\n"
|
||||
self.__data["comment"] += comment
|
||||
|
||||
def clearComments(self):
|
||||
self.__data["comment"] = ""
|
||||
|
||||
@staticmethod
|
||||
def columns():
|
||||
return InstallerStats.__FANCY_NAMES
|
||||
|
||||
def asRow(self):
|
||||
return [self.__data[k] for k in InstallerStats.__DATA.keys()]
|
||||
|
||||
def formattedDate(self):
|
||||
return datetime.datetime.fromtimestamp(int(self.peBuild)).strftime('%Y-%m-%d %H:%M:%S')
|
||||
|
||||
def formattedScriptDate(self):
|
||||
return datetime.datetime.fromtimestamp(int(self.scriptBuild)).strftime('%Y-%m-%d %H:%M:%S')
|
||||
|
||||
def gatherInfo(self):
|
||||
otherStats = rewise.getRewiseDevOtherStats(self.realpath)
|
||||
self.__data["peBuild"] = otherStats["peBuild"]
|
||||
self.__data["scriptBuild"] = otherStats["scriptBuild"]
|
||||
self.__data["isNe"] = 1 if otherStats["isNe"] else 0
|
||||
self.__data["isPk"] = 1 if otherStats["isPk"] else 0
|
||||
self.__data["overlay"] = otherStats["overlayOffset"]
|
||||
self.__data["scriptDeflateOffset"] = otherStats["scriptDeflateOffset"]
|
||||
self.__data["title"] = otherStats["title"]
|
||||
self.__data["langCount"] = otherStats["langCount"]
|
||||
|
||||
self.overlayHeaderUnk2 = otherStats["overlayUnk2"]
|
||||
self.wsHeaderUnk22 = otherStats["wsHdrUnk22"]
|
||||
|
||||
def testDevVerify(self):
|
||||
devVerify = rewise.getRewiseDevVerifyStats(self.realpath)
|
||||
self.__data["devStatus"] = rewise.getRewiseStatusString(devVerify["status"])
|
||||
self.__data["devError"] = devVerify["error"]
|
||||
|
||||
def testDevList(self):
|
||||
devList = rewise.getRewiseDevListStats(self.realpath)
|
||||
self.__data["devListStatus"] = rewise.getRewiseStatusString(devList["status"])
|
||||
#self.__data["devListError"] = devList["error"]
|
||||
|
||||
def testDevRaw(self):
|
||||
devRaw = rewise.getRewiseDevRawVerifyStats(self.realpath)
|
||||
self.__data["devRawStatus"] = rewise.getRewiseStatusString(devRaw["status"])
|
||||
self.__data["devRawError"] = devRaw["error"]
|
||||
|
||||
def test02Verify(self):
|
||||
r02Verify = rewise.getRewise02VerifyStats(self.realpath)
|
||||
self.__data["rew02Status"] = rewise.getRewiseStatusString(r02Verify["status"])
|
||||
self.__data["rew02Error"] = r02Verify["error"]
|
||||
|
||||
def generateComments(self):
|
||||
self.clearComments()
|
||||
analyze.addCustomComments(self)
|
||||
|
||||
if not self.isNe:
|
||||
if analyze.addPeComments(self):
|
||||
return
|
||||
|
||||
if self.overlayHeaderUnk2 and self.overlayHeaderUnk2 != "0008":
|
||||
self.addComment(f"Overlay header unknown2 != 0x8000 but 0x{self.overlayHeaderUnk2}.")
|
||||
|
||||
elif self.isPk:
|
||||
analyze.falsePositivePkCheck(self)
|
||||
|
||||
elif self.wsHeaderUnk22 and self.wsHeaderUnk22[-12:] not in ["000000000001", "000000000000"]:
|
||||
self.addComment(f"Possible weird stuff at WiseScriptHeader.unknown_22 '{self.wsHeaderUnk22[-12:]}'.")
|
||||
|
||||
elif self.scriptDeflateOffset != -1:
|
||||
if self.scriptDeflateOffset < -1:
|
||||
self.addComment(f"Negative script deflate offset")
|
||||
elif self.scriptDeflateOffset > 0x100000:
|
||||
self.addComment(f"Script deflate offset is absurd high")
|
||||
|
||||
def fullTest(self):
|
||||
self.gatherInfo()
|
||||
self.testDevVerify()
|
||||
self.testDevList()
|
||||
self.testDevRaw()
|
||||
self.test02Verify()
|
||||
|
||||
def installerStatsCsvIt(csvFile):
|
||||
with open(csvFile, newline='') as csvfile:
|
||||
spamreader = csv.reader(csvfile, delimiter=',', quotechar='"')
|
||||
for row in list(spamreader)[1:]: # skip the header
|
||||
yield InstallerStats(rowData=row)
|
||||
|
||||
|
||||
# Filter out installers that appear to be invalid / non Wise.
|
||||
# - When the installer is a PE file, it has a PE build date, filter
|
||||
# it out when the PE build date has no working installers.
|
||||
# - Filter out installer where the exe parsing failed.
|
||||
# - Filter out CRC32 corrupt installers.
|
||||
def acceptableInstallerIt(resultFile):
|
||||
installers = list(installerStatsCsvIt(resultFile))
|
||||
|
||||
# Get working PE dates (PE build dates with at least one success
|
||||
# fully operation for one installer).
|
||||
workingPeDates = []
|
||||
for installer in installers:
|
||||
if installer.peBuild in workingPeDates:
|
||||
continue
|
||||
|
||||
if (installer.devStatus == "OK" or
|
||||
installer.devRawStatus == "OK" or
|
||||
installer.devListStatus == "OK" or
|
||||
installer.rew02Status == "OK"):
|
||||
workingPeDates.append(installer.peBuild)
|
||||
|
||||
for installer in installers:
|
||||
if installer.devStatus in ["ERROR_INVALID_PENE", "ERROR_NOT_PENE",
|
||||
"ERROR_NOT_WISE"]:
|
||||
continue
|
||||
|
||||
if installer.peBuild not in workingPeDates:
|
||||
continue
|
||||
|
||||
if installer.devRawStatus == "ERROR_CRC32":
|
||||
# this installer is corrupt
|
||||
continue
|
||||
|
||||
yield installer
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
for source in sources.InstallerSourceUniqIt(os.path.join(SOURCE_PATH,
|
||||
"shareware.csv")):
|
||||
installer = InstallerStats(source)
|
||||
installer.gatherInfo()
|
||||
print(installer.title)
|
||||
250
tests/extended/results/README.md
Normal file
250
tests/extended/results/README.md
Normal file
|
|
@ -0,0 +1,250 @@
|
|||
# Results
|
||||
|
||||
These are the test results.
|
||||
|
||||
|
||||
## `shareware.csv`
|
||||
|
||||
> Use `python ../test.py print-stats shareware.csv` to print stats.
|
||||
|
||||
| DATE | OK | ERR | LOK | LERR | ROK | RERR | v02 | v02E | VERSION
|
||||
| :------------------ | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | :------
|
||||
| NE | 96 | 6 | 100 | 2 | 101 | 1 | 0 | 102 | Unknown
|
||||
| 1998-10-02 22:36:51 | 78 | 4 | 78 | 4 | 82 | 0 | 58 | 24 | Unknown
|
||||
| 1998-10-23 23:23:16 | 2 | 0 | 2 | 0 | 2 | 0 | 2 | 0 | Unknown
|
||||
| 1998-11-09 21:17:09 | 176 | 22 | 179 | 19 | 198 | 0 | 132 | 66 | InstallMaster 7
|
||||
| 1998-12-03 23:11:32 | 43 | 0 | 43 | 0 | 43 | 0 | 28 | 15 | Unknown
|
||||
| 1999-02-05 23:07:52 | 214 | 19 | 233 | 0 | 232 | 1 | 132 | 101 | Unknown
|
||||
| 1999-04-05 18:07:26 | 4 | 0 | 4 | 0 | 4 | 0 | 4 | 0 | Unknown
|
||||
| 1999-05-21 22:48:48 | 406 | 33 | 420 | 19 | 432 | 7 | 267 | 172 | Unknown
|
||||
| 1999-08-17 17:25:48 | 238 | 24 | 250 | 12 | 254 | 8 | 143 | 119 | Unknown
|
||||
| 2000-04-25 16:37:12 | 894 | 49 | 914 | 29 | 937 | 6 | 514 | 429 | InstallMaster 8
|
||||
| 2001-08-13 19:13:38 | 319 | 8 | 325 | 2 | 323 | 4 | 222 | 105 | Unknown
|
||||
| 2001-10-25 21:47:11 | 418 | 24 | 433 | 9 | 429 | 13 | 246 | 196 | Installation System 9
|
||||
| 2031-05-20 22:54:01 | 4 | 0 | 4 | 0 | 4 | 0 | 4 | 0 | Unknown
|
||||
| ------------------- | ---- | ---- | ---- | ---- | ---- | ---- | ---- | ---- | -------
|
||||
| Total: 3081 | 2892 | 189 | 2985 | 96 | 3041 | 40 | 1752 | 1329 |
|
||||
|
||||
```
|
||||
PK Total : 438
|
||||
PK OK : 432
|
||||
PK ERROR : 6
|
||||
PK Raw OK : 438
|
||||
PK Raw ERROR: 0
|
||||
|
||||
Comments:
|
||||
80 Without comment
|
||||
23 Negative script deflate offset
|
||||
21 Possible weird stuff at WiseScriptHeader.unknown_22 '9190A09A878B'.
|
||||
14 Possible weird stuff at WiseScriptHeader.unknown_22 '958A928F8C8B'.
|
||||
10 Possible weird stuff at WiseScriptHeader.unknown_22 'B890BE889E86'.
|
||||
6 Possible weird stuff at WiseScriptHeader.unknown_22 'B6BF88BF96BF'.
|
||||
6 Possible weird stuff at WiseScriptHeader.unknown_22 'A8B0ADB3BBB6'.
|
||||
5 'Initializing Wise Installation Wizard...' found in the PE b'.data\x00\x00\x00' section.
|
||||
3 Possible weird stuff at WiseScriptHeader.unknown_22 'BCADA6CEA5CD'.
|
||||
3 Possible weird stuff at WiseScriptHeader.unknown_22 'C8CACBC8C6CC'.
|
||||
2 Possible weird stuff at WiseScriptHeader.unknown_22 '99939E9D919E'.
|
||||
2 0x18 mismatch, 0x18 00 08 ..
|
||||
2 Possible weird stuff at WiseScriptHeader.unknown_22 'DB8F9E8C8C88'.
|
||||
2 Possible weird stuff at WiseScriptHeader.unknown_22 'ADBF90BF93BF'.
|
||||
1 Possible weird stuff at WiseScriptHeader.unknown_22 '8588C9C6C88B'.
|
||||
1 Possible weird stuff at WiseScriptHeader.unknown_22 'AB969C949A8B'.
|
||||
1 0x18 mismatch, there is OP 0x00 right after 0x18
|
||||
1 Script deflate offset is absurd high
|
||||
1 Possible weird stuff at WiseScriptHeader.unknown_22 'CC918B90929D'.
|
||||
1 Possible weird stuff at WiseScriptHeader.unknown_22 'B4AAACB6B5AC'.
|
||||
1 Possible weird stuff at WiseScriptHeader.unknown_22 '9D8D9E919B96'.
|
||||
1 Possible weird stuff at WiseScriptHeader.unknown_22 '9C909B9A989A'.
|
||||
1 Possible weird stuff at WiseScriptHeader.unknown_22 '8C929E9393D1'.
|
||||
1 Overlay header unknown2 != 0x8000 but 0x0000.
|
||||
|
||||
Verify errors:
|
||||
75 WS_ERROR_VALUE ERROR: parseWiseScriptHeader failed to read languageTexts.
|
||||
46 WS_ERROR_INFLATE ERROR: Inflated size is larger then expected
|
||||
25 WS_ERROR_SEEK ERROR: Failed seek to file offset
|
||||
9 ERROR_VALUE ERROR: Advertised WiseScript.bin size looks insane.
|
||||
7 WS_ERROR_INFLATE ERROR: zlib: invalid code lengths set
|
||||
4 WS_ERROR_INFLATE ERROR: zlib: invalid distance too far back
|
||||
4 WS_ERROR_INFLATE ERROR: zlib: invalid stored block lengths
|
||||
3 WS_ERROR_PK_HEADER ERROR: PK signature unknown
|
||||
3 WS_ERROR_VALUE ERROR: wiseScriptParsePath expected '%' at path start
|
||||
2 WS_ERROR_VALUE ERROR: parseWiseScript unknown OP: 3A
|
||||
2 WS_ERROR_INFLATE ERROR: zlib: invalid block type
|
||||
2 ERROR_VALUE ERROR: Advertised .dib size looks insane.
|
||||
1 WS_ERROR_BRANCHING ERROR: OP 0x23 LANG but not in LANG branch
|
||||
1 WS_ERROR_INFLATE ERROR: zlib: invalid literal/length code
|
||||
1 WS_ERROR_VALUE ERROR: This is a multi language installer but not all languages have their name set
|
||||
1 WS_ERROR_VALUE ERROR: parseWiseScript unknown OP: 49
|
||||
1 WS_ERROR_INFLATE ERROR: zlib: invalid distance code
|
||||
1 WS_ERROR_BRANCHING ERROR: OP 0x23 LANG but currently not in a branch
|
||||
1 ERROR_INFLATE ERROR: zlib: invalid stored block lengths
|
||||
```
|
||||
|
||||
## `ignored.csv`
|
||||
|
||||
> Use `python ../test.py print-stats ignored.csv` to print stats.
|
||||
|
||||
| DATE | OK | ERR | LOK | LERR | ROK | RERR | v02 | v02E | VERSION
|
||||
| :------------------ | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | :------
|
||||
| NE | 0 | 1 | 0 | 1 | 0 | 1 | 0 | 1 | Unknown
|
||||
| 1992-06-20 00:22:17 | 0 | 9 | 0 | 9 | 0 | 9 | 0 | 9 | Unknown
|
||||
| 1997-06-16 23:32:11 | 0 | 21 | 0 | 21 | 0 | 21 | 0 | 21 | Unknown
|
||||
| 1999-02-05 23:07:52 | 0 | 2 | 2 | 0 | 0 | 2 | 0 | 2 | Unknown
|
||||
| 1999-04-08 22:24:47 | 0 | 285 | 0 | 285 | 0 | 285 | 0 | 285 | Unknown
|
||||
| 1999-04-17 21:22:36 | 0 | 3 | 0 | 3 | 0 | 3 | 0 | 3 | Unknown
|
||||
| 1999-05-11 21:42:27 | 0 | 1 | 0 | 1 | 0 | 1 | 0 | 1 | Unknown
|
||||
| 1999-05-21 22:48:48 | 0 | 2 | 2 | 0 | 0 | 2 | 0 | 2 | Unknown
|
||||
| 1999-07-19 20:49:20 | 0 | 2 | 0 | 2 | 0 | 2 | 0 | 2 | Unknown
|
||||
| 1999-08-16 11:00:06 | 0 | 1 | 0 | 1 | 0 | 1 | 0 | 1 | Unknown
|
||||
| 1999-08-17 17:25:48 | 0 | 3 | 1 | 2 | 0 | 3 | 0 | 3 | Unknown
|
||||
| 1999-12-03 12:59:21 | 0 | 2 | 0 | 2 | 0 | 2 | 0 | 2 | Unknown
|
||||
| 2000-03-21 16:49:03 | 0 | 8 | 0 | 8 | 0 | 8 | 0 | 8 | Unknown
|
||||
| 2000-04-19 20:27:55 | 0 | 1 | 0 | 1 | 0 | 1 | 0 | 1 | Unknown
|
||||
| 2000-04-25 16:37:12 | 0 | 2 | 1 | 1 | 0 | 2 | 0 | 2 | InstallMaster 8
|
||||
| 2000-07-05 21:22:55 | 0 | 8 | 0 | 8 | 0 | 8 | 0 | 8 | Unknown
|
||||
| 2000-08-22 19:05:09 | 0 | 1 | 0 | 1 | 0 | 1 | 0 | 1 | Unknown
|
||||
| 2000-09-25 22:23:56 | 0 | 11 | 0 | 11 | 0 | 11 | 0 | 11 | Unknown
|
||||
| 2000-12-08 23:54:42 | 0 | 1 | 0 | 1 | 0 | 1 | 0 | 1 | Unknown
|
||||
| 2000-12-09 00:33:39 | 0 | 1 | 0 | 1 | 0 | 1 | 0 | 1 | Unknown
|
||||
| 2001-01-09 15:08:41 | 0 | 1 | 0 | 1 | 0 | 1 | 0 | 1 | Unknown
|
||||
| 2001-01-09 15:09:05 | 0 | 1 | 0 | 1 | 0 | 1 | 0 | 1 | Unknown
|
||||
| 2001-01-11 21:55:28 | 0 | 1 | 0 | 1 | 0 | 1 | 0 | 1 | Unknown
|
||||
| 2001-03-06 21:15:30 | 0 | 1 | 0 | 1 | 0 | 1 | 0 | 1 | Unknown
|
||||
| 2001-05-22 22:13:59 | 0 | 23 | 0 | 23 | 0 | 23 | 0 | 23 | Unknown
|
||||
| 2001-08-10 01:13:34 | 0 | 1 | 0 | 1 | 0 | 1 | 0 | 1 | Unknown
|
||||
| 2001-08-13 19:13:38 | 0 | 1 | 1 | 0 | 0 | 1 | 0 | 1 | Unknown
|
||||
| 2001-09-05 19:02:57 | 0 | 1 | 0 | 1 | 0 | 1 | 0 | 1 | Unknown
|
||||
| 2001-10-25 21:47:11 | 0 | 2 | 2 | 0 | 0 | 2 | 0 | 2 | Installation System 9
|
||||
| 2001-10-26 14:34:09 | 0 | 3 | 0 | 3 | 0 | 3 | 0 | 3 | Unknown
|
||||
| 2001-11-21 15:41:35 | 0 | 2 | 0 | 2 | 0 | 2 | 0 | 2 | Unknown
|
||||
| 2001-12-19 16:19:42 | 0 | 10 | 0 | 10 | 0 | 10 | 0 | 10 | Unknown
|
||||
| 2002-01-16 21:20:28 | 0 | 1 | 0 | 1 | 0 | 1 | 0 | 1 | Unknown
|
||||
| 2002-01-22 21:47:35 | 0 | 2 | 0 | 2 | 0 | 2 | 0 | 2 | Unknown
|
||||
| 2002-02-12 16:33:09 | 0 | 1 | 0 | 1 | 0 | 1 | 0 | 1 | Unknown
|
||||
| 2002-03-29 17:59:03 | 0 | 12 | 0 | 12 | 0 | 12 | 0 | 12 | Unknown
|
||||
| 2002-05-23 22:35:44 | 0 | 12 | 0 | 12 | 0 | 12 | 0 | 12 | Unknown
|
||||
| 2002-06-24 20:22:31 | 0 | 2 | 0 | 2 | 0 | 2 | 0 | 2 | Unknown
|
||||
| 2002-08-05 17:25:35 | 0 | 1 | 0 | 1 | 0 | 1 | 0 | 1 | Unknown
|
||||
| 2002-11-18 16:53:37 | 0 | 1 | 0 | 1 | 0 | 1 | 0 | 1 | Unknown
|
||||
| 2002-12-02 19:31:43 | 0 | 2 | 0 | 2 | 0 | 2 | 0 | 2 | Unknown
|
||||
| 2002-12-16 20:16:07 | 0 | 17 | 0 | 17 | 0 | 17 | 0 | 17 | Unknown
|
||||
| 2003-01-20 15:36:28 | 0 | 1 | 0 | 1 | 0 | 1 | 0 | 1 | Unknown
|
||||
| 2003-02-03 05:29:30 | 0 | 1 | 0 | 1 | 0 | 1 | 0 | 1 | Unknown
|
||||
| 2003-03-12 11:26:56 | 0 | 1 | 0 | 1 | 0 | 1 | 0 | 1 | Unknown
|
||||
| 2003-04-14 17:47:48 | 0 | 6 | 0 | 6 | 0 | 6 | 0 | 6 | Unknown
|
||||
| 2003-04-22 18:23:04 | 0 | 1 | 0 | 1 | 0 | 1 | 0 | 1 | Unknown
|
||||
| 2003-04-24 15:10:23 | 0 | 4 | 0 | 4 | 0 | 4 | 0 | 4 | Unknown
|
||||
| 2003-05-02 22:34:49 | 0 | 1 | 0 | 1 | 0 | 1 | 0 | 1 | Unknown
|
||||
| 2003-07-07 17:37:27 | 0 | 11 | 0 | 11 | 0 | 11 | 0 | 11 | Unknown
|
||||
| 2003-10-09 21:05:12 | 0 | 1 | 0 | 1 | 0 | 1 | 0 | 1 | Unknown
|
||||
| 2003-11-21 19:16:37 | 0 | 23 | 0 | 23 | 0 | 23 | 0 | 23 | Unknown
|
||||
| 2003-12-09 07:17:05 | 0 | 3 | 0 | 3 | 0 | 3 | 0 | 3 | Unknown
|
||||
| 2004-01-29 07:13:04 | 0 | 20 | 0 | 20 | 0 | 20 | 0 | 20 | Unknown
|
||||
| 2004-04-19 06:44:28 | 0 | 1 | 0 | 1 | 0 | 1 | 0 | 1 | Unknown
|
||||
| 2004-06-10 17:30:49 | 0 | 1 | 0 | 1 | 0 | 1 | 0 | 1 | Unknown
|
||||
| 2004-07-16 08:08:27 | 0 | 2 | 0 | 2 | 0 | 2 | 0 | 2 | Unknown
|
||||
| 2004-08-12 15:48:03 | 0 | 9 | 0 | 9 | 0 | 9 | 0 | 9 | Unknown
|
||||
| 2004-09-10 00:10:25 | 0 | 1 | 0 | 1 | 0 | 1 | 0 | 1 | Unknown
|
||||
| 2004-09-10 17:38:04 | 0 | 2 | 0 | 2 | 0 | 2 | 0 | 2 | Unknown
|
||||
| 2004-10-22 09:20:02 | 0 | 2 | 0 | 2 | 0 | 2 | 0 | 2 | Unknown
|
||||
| 2004-10-28 13:15:26 | 0 | 1 | 0 | 1 | 0 | 1 | 0 | 1 | Unknown
|
||||
| 2004-12-10 17:05:37 | 0 | 1 | 0 | 1 | 0 | 1 | 0 | 1 | Unknown
|
||||
| 2005-04-08 16:46:17 | 0 | 7 | 0 | 7 | 0 | 7 | 0 | 7 | Unknown
|
||||
| 2005-08-16 01:42:29 | 0 | 1 | 0 | 1 | 0 | 1 | 0 | 1 | Unknown
|
||||
| 2005-12-13 23:12:15 | 0 | 40 | 0 | 40 | 0 | 40 | 0 | 40 | Unknown
|
||||
| 2006-11-13 19:32:07 | 0 | 2 | 0 | 2 | 0 | 2 | 0 | 2 | Unknown
|
||||
| 2006-11-27 18:36:03 | 0 | 1 | 0 | 1 | 0 | 1 | 0 | 1 | Unknown
|
||||
| 2006-12-15 22:21:24 | 0 | 4 | 0 | 4 | 0 | 4 | 0 | 4 | Unknown
|
||||
| 2007-01-20 06:07:04 | 0 | 1 | 0 | 1 | 0 | 1 | 0 | 1 | Unknown
|
||||
| 2007-01-29 21:17:00 | 0 | 1 | 0 | 1 | 0 | 1 | 0 | 1 | Unknown
|
||||
| 2007-03-26 16:49:41 | 0 | 1 | 0 | 1 | 0 | 1 | 0 | 1 | Unknown
|
||||
| 2007-08-16 20:13:16 | 0 | 8 | 0 | 8 | 0 | 8 | 0 | 8 | Unknown
|
||||
| 2007-11-20 22:12:45 | 0 | 2 | 0 | 2 | 0 | 2 | 0 | 2 | Unknown
|
||||
| 2007-11-20 22:52:34 | 0 | 9 | 0 | 9 | 0 | 9 | 0 | 9 | Unknown
|
||||
| 2008-06-25 11:42:41 | 0 | 1 | 0 | 1 | 0 | 1 | 0 | 1 | Unknown
|
||||
| 2009-10-27 17:10:03 | 0 | 1 | 0 | 1 | 0 | 1 | 0 | 1 | Unknown
|
||||
| 2010-03-10 00:20:10 | 0 | 1 | 0 | 1 | 0 | 1 | 0 | 1 | Unknown
|
||||
| 2011-07-04 12:17:32 | 0 | 2 | 0 | 2 | 0 | 2 | 0 | 2 | Unknown
|
||||
| 2012-02-17 00:42:19 | 0 | 1 | 0 | 1 | 0 | 1 | 0 | 1 | Unknown
|
||||
| 2012-03-20 15:00:47 | 0 | 2 | 0 | 2 | 0 | 2 | 0 | 2 | Unknown
|
||||
| 2012-10-29 18:13:37 | 0 | 1 | 0 | 1 | 0 | 1 | 0 | 1 | Unknown
|
||||
| 2012-12-07 18:34:27 | 0 | 1 | 0 | 1 | 0 | 1 | 0 | 1 | Unknown
|
||||
| 2013-03-28 11:38:55 | 0 | 1 | 0 | 1 | 0 | 1 | 0 | 1 | Unknown
|
||||
| ------------------- | ---- | ---- | ---- | ---- | ---- | ---- | ---- | ---- | -------
|
||||
| Total: 644 | 0 | 644 | 9 | 635 | 0 | 644 | 0 | 644 |
|
||||
|
||||
```
|
||||
PK Total : 152
|
||||
PK OK : 0
|
||||
PK ERROR : 152
|
||||
PK Raw OK : 0
|
||||
PK Raw ERROR: 152
|
||||
|
||||
Comments:
|
||||
10 Without comment
|
||||
329 'Initializing Wise Installation Wizard...' found in the PE b'.rsrc\x00\x00\x00' section.
|
||||
181 'Initializing Wise Installation Wizard...' found in the PE b'.rdata\x00\x00' section.
|
||||
41 No overlay data
|
||||
16 Overlay header unknown2 != 0x8000 but 0x0000.
|
||||
10 'Initializing Wise Installation Wizard...' found in the PE b'.WISE\x00\x00\x00' section.
|
||||
4 'Initializing Wise Installation Wizard...' found in the PE b'_winzip_' section.
|
||||
3 Overlay header unknown2 != 0x8000 but 0x3137.
|
||||
3 Overlay header unknown2 != 0x8000 but 0x496E.
|
||||
2 Overlay header unknown2 != 0x8000 but 0x481C.
|
||||
2 Overlay header unknown2 != 0x8000 but 0x5468.
|
||||
2 Overlay header unknown2 != 0x8000 but 0x6973.
|
||||
2 Overlay header unknown2 != 0x8000 but 0x003E.
|
||||
2 Overlay header unknown2 != 0x8000 but 0x8620.
|
||||
2 Possible weird stuff at WiseScriptHeader.unknown_22 '958A928F8C8B'.
|
||||
2 Overlay header unknown2 != 0x8000 but 0x312C.
|
||||
1 This not a PE file: 'DOS Header magic not found.'
|
||||
1 Overlay header unknown2 != 0x8000 but 0x383E.
|
||||
1 Overlay header unknown2 != 0x8000 but 0x6233.
|
||||
1 Overlay header unknown2 != 0x8000 but 0x7376.
|
||||
1 Overlay header unknown2 != 0x8000 but 0x3335.
|
||||
1 Overlay header unknown2 != 0x8000 but 0x0205.
|
||||
1 Overlay header unknown2 != 0x8000 but 0x0355.
|
||||
1 Overlay header unknown2 != 0x8000 but 0x2E31.
|
||||
1 Very small overlay size of 3496 bytes
|
||||
1 Overlay header unknown2 != 0x8000 but 0xE819.
|
||||
1 Overlay header unknown2 != 0x8000 but 0x0713.
|
||||
1 Overlay header unknown2 != 0x8000 but 0x4C49.
|
||||
1 Overlay header unknown2 != 0x8000 but 0x0062.
|
||||
1 Overlay header unknown2 != 0x8000 but 0xDF64.
|
||||
1 False positive PK, check the overlay header
|
||||
1 Overlay header unknown2 != 0x8000 but 0x7072.
|
||||
1 Overlay header unknown2 != 0x8000 but 0x6974.
|
||||
1 Overlay header unknown2 != 0x8000 but 0x6520.
|
||||
1 Overlay header unknown2 != 0x8000 but 0x7365.
|
||||
1 Overlay header unknown2 != 0x8000 but 0x0700.
|
||||
1 Overlay header unknown2 != 0x8000 but 0x74CD.
|
||||
1 Overlay header unknown2 != 0x8000 but 0x0787.
|
||||
1 Overlay header unknown2 != 0x8000 but 0x1100.
|
||||
1 Overlay header unknown2 != 0x8000 but 0xB611.
|
||||
1 Overlay header unknown2 != 0x8000 but 0x7E20.
|
||||
1 Overlay header unknown2 != 0x8000 but 0x4544.
|
||||
1 Overlay header unknown2 != 0x8000 but 0xA21E.
|
||||
1 Overlay header unknown2 != 0x8000 but 0x6173.
|
||||
1 Overlay header unknown2 != 0x8000 but 0x0AAF.
|
||||
1 Overlay header unknown2 != 0x8000 but 0x726F.
|
||||
1 Overlay header unknown2 != 0x8000 but 0xD753.
|
||||
1 Overlay header unknown2 != 0x8000 but 0xE50A.
|
||||
1 Overlay header unknown2 != 0x8000 but 0x6F67.
|
||||
|
||||
Verify errors:
|
||||
354 ERROR_VALUE ERROR: Advertised WiseScript.bin size looks insane.
|
||||
227 ERROR_INVALID_PENE ERROR: analyzeExe overlay size to small.
|
||||
28 ERROR_VALUE ERROR: Advertised .dib size looks insane.
|
||||
15 ERROR_INFLATE ERROR: zlib: invalid stored block lengths
|
||||
4 WS_ERROR_INFLATE ERROR: Inflated size is larger then expected
|
||||
4 WS_ERROR_CRC32 ERROR: CRC32 mismatch!
|
||||
2 ERROR_INFLATE ERROR: zlib: invalid distance too far back
|
||||
2 WS_ERROR_VALUE ERROR: parseWiseScriptHeader failed to read languageTexts.
|
||||
2 ERROR_INFLATE ERROR: zlib: invalid block type
|
||||
1 ERROR_INVALID_PENE ERROR: analyzeExe this is not a PE file for sure. The MS-DOS header signature doesn't match (not MZ).
|
||||
1 ERROR_INVALID_PENE ERROR: analyzeExe overlay offset larger then file size.
|
||||
1 WS_ERROR_SYSTEM_IO ERROR: Failed to find PK central directory size
|
||||
1 WS_ERROR_CRC32 ERROR: PK CRC32 mismatch for '(null)'
|
||||
1 ERROR_INFLATE ERROR: zlib: invalid code lengths set
|
||||
1 ERROR_INFLATE ERROR: zlib: invalid distance code
|
||||
```
|
||||
3747
tests/extended/results/ignored.csv
Normal file
3747
tests/extended/results/ignored.csv
Normal file
File diff suppressed because it is too large
Load diff
7288
tests/extended/results/shareware.csv
Normal file
7288
tests/extended/results/shareware.csv
Normal file
File diff suppressed because it is too large
Load diff
157
tests/extended/rewise.py
Normal file
157
tests/extended/rewise.py
Normal file
|
|
@ -0,0 +1,157 @@
|
|||
import subprocess
|
||||
|
||||
from config import REWISE_DEV_PATH, REWISE_02_PATH
|
||||
|
||||
|
||||
REWISE_STATUS_UNSET = -1
|
||||
REWISE_STATUS_OK = 0
|
||||
|
||||
|
||||
def getRewiseStatusString(status):
|
||||
wsError = status & 64
|
||||
status -= wsError
|
||||
|
||||
statusMap = {
|
||||
-1: "UNSET",
|
||||
0: "OK",
|
||||
1: "ERROR_SYSTEM_IO",
|
||||
2: "ERROR_ARGS",
|
||||
3: "ERROR_INTERNAL",
|
||||
4: "ERROR_EOF",
|
||||
5: "ERROR_SEEK",
|
||||
6: "ERROR_VALUE",
|
||||
8: "ERROR_INVALID_PENE",
|
||||
9: "ERROR_NOT_PENE",
|
||||
10: "ERROR_NOT_WISE",
|
||||
16: "ERROR_INFLATE",
|
||||
17: "ERROR_PK_HEADER",
|
||||
18: "ERROR_CRC32",
|
||||
24: "ERROR_BRANCHING"
|
||||
}
|
||||
|
||||
if status in statusMap:
|
||||
if wsError:
|
||||
return f"WS_{statusMap[status]}"
|
||||
return statusMap[status]
|
||||
else:
|
||||
return "SEGFAULT"
|
||||
|
||||
|
||||
PARSE_TYPE_INT = 1 # parse first int after text
|
||||
PARSE_TYPE_STR = 2 # parse text after match
|
||||
PARSE_TYPE_MATCH = 3 # bool (True when match found)
|
||||
|
||||
DEV_STDERR_PARSE = {
|
||||
#attr name parse type match default
|
||||
"isNe" : (PARSE_TYPE_MATCH, "DEBUG: NE ", False),
|
||||
"isPk" : (PARSE_TYPE_MATCH, "DEBUG: IsPkFormat: 1", False),
|
||||
"overlayOffset": (PARSE_TYPE_INT, "DEBUG: OverlayOffset: ", -1),
|
||||
"peBuild" : (PARSE_TYPE_INT, "DEBUG: PE Build: ", 0),
|
||||
"langCount" : (PARSE_TYPE_INT, "DEBUG: LanguageCount: ", -1),
|
||||
"overlayUnk2" : (PARSE_TYPE_STR, "OverlayHeader.unknown_2 : ", ""),
|
||||
"scriptDeflateOffset": (PARSE_TYPE_INT, "DEBUG: scriptDeflateOffset: ", -1)
|
||||
}
|
||||
|
||||
DEV_STDOUT_PARSE = {
|
||||
"wsHdrUnk22" : (PARSE_TYPE_STR, "WiseScriptHeader.unknown_22 : ", ""),
|
||||
"scriptBuild" : (PARSE_TYPE_INT, "WiseScriptHeader.datetime : ", -1)
|
||||
}
|
||||
|
||||
|
||||
def getLinesFromBytes(the_bytes):
|
||||
for line in the_bytes.decode("utf-8", errors="replace").split("\n"):
|
||||
yield line
|
||||
|
||||
|
||||
def parse(lines, parsedict):
|
||||
result = {}
|
||||
for line in lines:
|
||||
for opt in parsedict:
|
||||
ptype = parsedict[opt][0]
|
||||
tomatch = parsedict[opt][1]
|
||||
|
||||
if line.startswith(tomatch):
|
||||
if ptype == PARSE_TYPE_INT:
|
||||
result.update({opt: int(line[len(tomatch):].split(" ")[0])})
|
||||
elif ptype == PARSE_TYPE_STR:
|
||||
result.update({opt: line[len(tomatch):].rstrip()})
|
||||
elif ptype == PARSE_TYPE_MATCH:
|
||||
result.update({opt: True})
|
||||
break
|
||||
|
||||
for opt in parsedict:
|
||||
if opt not in result:
|
||||
result.update({opt: parsedict[opt][2]})
|
||||
|
||||
return result
|
||||
|
||||
|
||||
def genericRewiseStats(file, rewisePath, rewiseOp):
|
||||
r = subprocess.run([rewisePath, rewiseOp, file],
|
||||
capture_output=True,
|
||||
text=True, encoding="utf8")
|
||||
|
||||
result = {
|
||||
"status": REWISE_STATUS_UNSET,
|
||||
"error": ""
|
||||
}
|
||||
|
||||
if r.returncode == 0:
|
||||
result["status"] = REWISE_STATUS_OK
|
||||
elif r.returncode < 127:
|
||||
result["status"] = r.returncode
|
||||
|
||||
error = ""
|
||||
errCount = 0
|
||||
for line in reversed(r.stderr.split("\n")):
|
||||
if line.startswith("ERROR:"):
|
||||
error += line + "\n"
|
||||
errCount += 1
|
||||
if errCount == 5:
|
||||
break;
|
||||
result["error"] = error # last lines
|
||||
else:
|
||||
result["status"] = r.returncode
|
||||
|
||||
return (r, result)
|
||||
|
||||
|
||||
def getRewiseDevVerifyStats(file):
|
||||
return genericRewiseStats(file, REWISE_DEV_PATH, "-Vd")[1]
|
||||
|
||||
def getRewiseDevListStats(file):
|
||||
return genericRewiseStats(file, REWISE_DEV_PATH, "-l")[1]
|
||||
|
||||
def getRewiseDevRawVerifyStats(file):
|
||||
return genericRewiseStats(file, REWISE_DEV_PATH, "--raw-verify")[1]
|
||||
|
||||
def getRewise02VerifyStats(file):
|
||||
return genericRewiseStats(file, REWISE_02_PATH, "-V")[1]
|
||||
|
||||
def getRewiseDevOtherStats(file):
|
||||
r = subprocess.run([REWISE_DEV_PATH, "-ld", file],
|
||||
capture_output=True,
|
||||
text=False)
|
||||
result = {"title": ""}
|
||||
result.update(parse(getLinesFromBytes(r.stderr), DEV_STDERR_PARSE))
|
||||
|
||||
# This is extra, parse the title (inside WiseScript.bin)
|
||||
r = subprocess.run([REWISE_DEV_PATH, "-zd", file],
|
||||
capture_output=True,
|
||||
text=False)
|
||||
textsStart = False
|
||||
for line in getLinesFromBytes(r.stdout):
|
||||
if textsStart and line.startswith("Text: "):
|
||||
result.update({"title": line[6:].replace("\"", "")})
|
||||
break
|
||||
if line.startswith("WiseScript Texts"):
|
||||
textsStart = True
|
||||
|
||||
result.update(parse(getLinesFromBytes(r.stdout), DEV_STDOUT_PARSE))
|
||||
|
||||
return result
|
||||
|
||||
# For mass grepping
|
||||
def debugPrint(file):
|
||||
subprocess.run([REWISE_DEV_PATH, "-zd", file],
|
||||
capture_output=False)
|
||||
53
tests/extended/sources.py
Normal file
53
tests/extended/sources.py
Normal file
|
|
@ -0,0 +1,53 @@
|
|||
import csv
|
||||
import os
|
||||
|
||||
from config import SOURCE_PATH, DL_PATH
|
||||
|
||||
|
||||
""" -- InstallerSource
|
||||
|
||||
Read sources to Wise installers.
|
||||
|
||||
The file format is CSV with the following columns
|
||||
(b3, md5, filesize, url). It does not have a header.
|
||||
"""
|
||||
|
||||
class InstallerSource:
|
||||
def __init__(self, data):
|
||||
self.b3, self.md5, self.filesize, self.dlUrl = data
|
||||
self.filesize = int(self.filesize)
|
||||
self.filepath = os.path.join(DL_PATH, self.b3)
|
||||
|
||||
def getFilename(self):
|
||||
return os.path.basename(self.dlUrl)
|
||||
|
||||
|
||||
def InstallerSourceIt(swFile):
|
||||
with open(swFile, newline='') as fp:
|
||||
for row in csv.reader(fp):
|
||||
yield InstallerSource(row)
|
||||
|
||||
|
||||
def InstallerSourceUniqIt(swFile):
|
||||
sums = []
|
||||
with open(swFile, newline='') as fp:
|
||||
for row in csv.reader(fp):
|
||||
sw = InstallerSource(row)
|
||||
if sw.b3 in sums:
|
||||
continue
|
||||
sums.append(sw.b3)
|
||||
yield sw
|
||||
|
||||
|
||||
def InstallerSourceAllUniqIt():
|
||||
for ent in os.listdir(SOURCE_PATH):
|
||||
abspath = os.path.join(SOURCE_PATH, ent)
|
||||
if not os.path.isfile(abspath):
|
||||
continue
|
||||
|
||||
if os.path.splitext(ent)[1] != ".csv":
|
||||
continue
|
||||
|
||||
for source in InstallerSourceUniqIt(abspath):
|
||||
yield source
|
||||
|
||||
27
tests/extended/sources/README.md
Normal file
27
tests/extended/sources/README.md
Normal file
|
|
@ -0,0 +1,27 @@
|
|||
# sources
|
||||
|
||||
`.exe` files from <http://cd.textfiles.com>,
|
||||
<http://discmaster.textfiles.com> and <https://archive.org> have been
|
||||
grepped for the `Initializing Wise Installation Wizard...` string. The
|
||||
files here are the results.
|
||||
|
||||
There are duplicates since every URL found is listed.
|
||||
|
||||
All the `.csv` files here have the same `b3, md5, filesize, dl-url`
|
||||
structure, without header.
|
||||
|
||||
## `shareware.csv`
|
||||
|
||||
- Total size is 70.19 GiB
|
||||
|
||||
|
||||
## `ignored.csv`
|
||||
|
||||
- Total size is 17.08 GiB
|
||||
|
||||
These have been filtered out, they are assumed false positives.
|
||||
|
||||
See the `acceptableInstallerIt()` function in `../installers.py` for the
|
||||
filter rules.
|
||||
|
||||
|
||||
1175
tests/extended/sources/ignored.csv
Normal file
1175
tests/extended/sources/ignored.csv
Normal file
File diff suppressed because it is too large
Load diff
4596
tests/extended/sources/shareware.csv
Normal file
4596
tests/extended/sources/shareware.csv
Normal file
File diff suppressed because it is too large
Load diff
669
tests/extended/test.py
Normal file
669
tests/extended/test.py
Normal file
File diff suppressed because it is too large
Load diff
Loading…
Reference in a new issue