Draft HardPressed Documentation Andy McFadden Revised: 18-Mar-93 (touched up a bit 1998/06/06) I. Intro stuff 1. Congratulations on your purchase - feature list - minimum requirements (1MB RAM, 2MB+HD recommended - basically the same stuff as for System 6.0) 2. How to turn the power on, put the disk in, and install it II. About HardPressed 1. Basic operating principles HardPressed works by compressing and uncompressing individual files. Unlike products for other machines (e.g. The Stacker), you don't need to create special partitions for the compressed data. All of your files will continue to live in the same places. Data compression is a CPU-intensive operation. By compressing files you are trading off disk space for time. In most cases, access to compressed files will be slower than it would be if the files weren't compressed. Access may actually be faster in certain cases, such as when you use a fast compression method on a slow disk device such as an AppleDisk 3.5" or 5.25" drive. You should not notice any reduction in speed when accessing uncompressed files. 1a. compress me a file With HardPressed, files can be compressed manually or automatically. Manual compression is done by selecting the files from the Finder and using the HardPressed Finder(tm) Extension to compress or uncompress them. Automatic compression is done by "marking" folders so that files which are created or changed inside them are automatically compressed. The marks can be added or removed from the CDev. You can also control whether or not ANY file is automatically compressed by selecting one of the following modes: (1) Compress and Expand. Files will be expanded when accessed, and files in folders which have been "marked" will be compressed automatically when they are created or changed. (2) Expand only. Files will be expanded when accessed, but will not be re-compressed after they are created or changed. (3) Inactive. Files will not be compressed or expanded. In addition, they will appear as HardPressed files in the Finder instead of their actual type. The mode can be changed from the CDev or the CDA. To specify how you want the files to be compressed, you may choose one of several compression "profiles", which allow you to select greater compression at a loss of speed, or greater speed with less compression. These are chosen from the CDev. If you want to be sure that the file has been compressed correctly, you can choose a "verify" option which will automatically test the compressed version before removing the original. 2. How to use HardPressed HardPressed has four components: an INIT, a CDev (Control Panel), a CDA (Classic Desk Accessory), and a Finder Extension. The INIT is the workhorse of the system. It performs the actual compression and uncompression of files. Because it works invisibly, you don't need to interact with it; however, if it isn't around, none of the other components will function. HardPressed will not compress anything until you tell it to. Simply installing HardPressed will not cause any compression to occur. There are two ways to specify files, one manual and one automatic. 2a. The FExt You can use the Finder(tm) extension to manually select files to compress and expand. To use it, select the files you want to compress from the Finder, and choose the "Compress" item from the "Extras" menu. A dialog similar to the "copying files" dialog [picture?] will inform you of your progress. You can select files, folders, or entire disks. The disks can be ProDOS, HFS, or any other format that GS/OS can write to. The "Expand" option works similarly, but in reverse. Use this if you decide that a file, folder, or disk should no longer be compressed. It is NOT necessary to expand files to use them from GS/OS! (That would defeat the purpose.) You may find it useful to compress and expand ProDOS 8 files before using them; this can serve as a convenient way to archive and extract them. To see how much space you have saved, select some files and choose the "Compression Info" item. The dialog will tell you the average compression of the compressed files, and how much space was saved overall. In addition, if you only select one file, the method of compression used will be displayed. Bear in mind that the savings here may not match the actual reduction in disk usage. There are two big factors here: (1) on most disks that GS/OS supports, space is allocated in 512 byte chunks. This means that, if you only have 511 bytes of data in a file, no amount of compression is going to reduce the amount of disk space. On the other hand, if you have a 513 byte file, reducing the size by one byte will cut the space needed in half. (2) ProDOS 8 and the GS/OS ProDOS FST (File System Translator) support "sparse" files. These are files which are often huge but almost entirely empty. When GS/OS (more accurately, the ProDOS FST) goes to write a block, it first checks to see if the block is full of zeros. If so, it just makes a note that the block is empty, and doesn't allocate one. Thus, a 100K file might only use 10K of space on the disk, and while compression might reduce this to 5K, you will only have a 50% reduction in disk usage even though the file itself was compressed by 95%. By and large, however, the results will be accurate to within a few percent. HardPressed will only compress and expand "locked" files if the appropriate box in the CDev's "Preferences" screen is checked. Sometimes HardPressed will think a file is locked even though the Finder doesn't display it as such (I consider this a bug in the Finder). If you seem to be banging your mouse against the wall and getting nowhere, try locking and unlocking the file from the Finder's "File Info" dialog (click on the file and hit OA-I). 2b. The CDev The other way to tell HardPressed to compress your files is from the CDev. The CDev has a number of other features, so it's probably best to describe it frame-by-frame. [ picture of initial screen ] You may choose the mode (Compress and Expand, Expand only, and Inactive) by clicking on the appropriate button. IMPORTANT: Because of limitations in the Finder, it is not possible to redraw the files after switching the system to or from "Inactive" mode. For the icons to appear differently, the window must be closed and then reopened. [ can we do this automatically? ] It is recommended that you do not leave the system in Inactive mode to avoid confusion. The Profile pop-up menu displays a list of profiles found in the "HP.Modules" folder (which in turn is in your */System/System.Setup folder). Choosing one with the mouse will cause that profile to be loaded. If you want to return to the Standard profile, you can just hit Open-Apple-S. The "Preferences..." button takes you to the preferences screen. You can configure several different aspects of HardPressed from here. By clicking on the "Verify" box, you are enabling verify mode. This causes HardPressed to validate the compressed version of the file before removing the original. This requires that the compressed version be expanded, so selecting this option may cause operations to take twice as long. The "show progress with mouse cursor" [what'd I call it?] button lets you decide whether or not the mouse cursor should change to a filling circle while HP is compressing. The Disk Caching pop-up menu lets you tell HardPressed how much memory it should use to cache the files it is working on. This value is a maximum limit; HardPressed does NOT permanently take away that much memory from the system. If you set the cache size to "Off", no caching will be done. The lower half of the screen has the Finder Extension preferences. You can choose whether or not locked files will be compressed and expanded, and can choose which keys to use for the FExt commands. The "Temp. Folder..." button brings up a small window that lets you tell HardPressed how and where you want temporary files to be stored. When compressing or expanding, HardPressed needs to have a copy of both the expanded and compressed versions of the file. If it can't fit in the disk cache, HardPressed will create a temporary file. If you don't check the "Use a single folder" box, the temporary files will be stored in the same folder as the file being compressed or expanded. If you do check the box, all of the temporary files will be stored in a single folder. This is the recommended option, because it lets you choose the fastest device you have (such as a RAM disk or fast hard drive), and allows you to work with write-protected disks. The disk used to hold the temporary files should have at least enough free space to hold the largest file you plan to compress or expand. You can select the folder in one of two ways. The first (and recommended) method is to press the "Select..." button [key equiv OA-S... actually, there are key equivs for just about everything]. This brings up a standard file dialog; move to the appropriate folder and press "Accept". The second way is to just type the name in. However, very little error checking is performed, so if you make a typo HardPressed will be unable to create temporary files. The lack of error checking allows you to make mistakes, but it also allows you to do unusual things. For example, the default temporary file folder is "*:", which is shorthand for "use the volume directory of whatever disk the system was booted from." The "Folders..." button takes you to the disk selection screen. You will be presented with a list of the disks that HardPressed was able to find. Opening one of them (by double-clicking on the name or selecting Open) takes you to the folder selection screen. You can switch to another disk at any time by selecting the disk from the pop-up menu at the top. Here you will see a list of the folders present on the disk you chose. By using Open and Close [key equivs OA-O, OA-down-arrow, Esc, OA-up-arrow], you can open up or close the folders. To mark a folder so that the files in it will automatically be compressed after being created or changed, select the folder from the list and hit the Mark button [M, OA-M, space toggles]. You can unmark it with the UnMark button [U, OA-U, space]. An important feature of the folder selection screen is that what you see is what you get. If a folder does not appear on the screen, then its mark will be the same as the mark for its parent. So if you have the following: :MyDisk Applications X Data Drawing X WordProcessor Games Then only files that are in :MyDisk:Data and :MyDisk:Data:WordProcessor will be compressed automatically. Files in :MyDisk:Data:Drawing will NOT be, because the "Drawing" folder doesn't have a mark next to it. If on the other hand you had this (for the same disk, but the "Data" folder is now closed): :MyDisk Applications X Data Games Then everything under "Data" will be compressed, including those in "Drawing" and "WordProcessor". To reduce clutter, when you hit the "Done" button or switch to another disk, the HardPressed CDev will automatically close any folder whose children all have the same mark status. So if you had: :MyDisk Applications X Data X Drawing X WordProcessor Games Then the "Data" folder would be closed, leaving: :MyDisk Applications X Data Games Since the mark on "Data" affects all of the subdirectories, it isn't necessary to explicitly mark "Drawing" and "WordProcessor". After you mark some folders, the result will be saved in a file called "HP.Vol.Prefs" on the disk in question. The file will be invisible like the Finder's data files, so you will only see it if you set the Finder preference which allows you to see the Finder's data files. Pressing the "Revert" button will throw out any changes you have made, and re-read the information from the disk. Your changes are saved whenever you switch to a different disk with the pop-up menu or press the "Done" button. It is possible to select more than one folder at a time for any of these operations [standard list selection stuff, OA-A for select all]. To expedite the process of opening all the folders inside a selected folder, newly-opened folders will appear selected in the list. They will also have the mark status of the folder they are in. When you close the CDev, your preferences will be saved in a file called "HP.Prefs" on the boot disk in the "System.Setup" folder. To avoid the possibility of not being able to boot because some system files were compressed, the CDev will NOT save "Inactive" status to the preferences file. If you close the CDev while HardPressed is Inactive, you will be in "Expand only" mode when you reboot. If you want to boot the system without HardPressed, press 'h' right after you see the GS/OS boot thermometer. The CDev's icon should appear with a red 'X' over it. 2c. CDA The CDA (Classic Desk Accessory) provides a quick interface to the boxed items on the CDev screen. You can change the mode to Compress and Expand, Expand only, or Inactive, and you can toggle Verify on and off. Changes made here do NOT affect the HP preferences file, so this is a good way to change settings temporarily. The CDA controls will not be accessible from ProDOS 8. 2d. Notes & random stuff It is important to remember that files will be recompressed ONLY if the folder they are in is marked. This means that if you change a compressed file in a folder that is marked as "Expand only", the file will be stored without compression. This has some advantages, which are explained later. IMPORTANT: the Finder extension will ignore the on/off switch and will override your folder settings. This is based on the assumption that you have deliberately chosen to compress or expand the files you have selected. This allows you to compress files without having to mark the folder they're in, which is useful for files which don't change. For example, you might have some sample source code or old database files that you want to reference, but you don't want the stuff you're working on to be compressed. You can leave the folder unmarked and compress the sample files from the Finder; that way you get the benefits of compression on the old files but nothing you edit or create will be compressed. This is especially useful for compressing stuff with schemes which compress slowly but extract very quickly, such as LZSS. Other examples include TrueType fonts, desk accessories, and applications. TIP: if you're working with some files in a marked folder, but you decide at the last minute that don't want them to be compressed, you don't need to go into the CDev to turn off the mark status. Instead, use the CDA to change the mode to "Expand only" until you are done. That way, none of the files will be compressed automatically, regardless of the folder mark status. HardPressed will always expand files that it has compressed, whether or not the folder they are in is marked. In addition, the current profile setting does not affect expansion. However, if you change the mode to Inactive, it won't expand anything. This can be useful when you want to copy a large collection of compressed files from one place to another. If you tried it the usual way, it would expand the files, and then either store them without compression or re-compress them on the destination disk. By first putting HardPressed into Inactive mode, you can copy the files as compressed data, which is considerably faster. (This is NOT recommended, of course, if you are copying the files onto a disk to give to somebody who - for whatever reason - doesn't own a copy of HardPressed.) The HardPressed compression code is stored in individual "modules", which are kept in the "HP.Modules" folder under "System.Setup". These can be inactivated from the Finder's "File Info" menu item if you don't want them to be loaded. This can be useful for infrequently used modules. For example, the adaptive Huffman and differential compression modules are inactive by default). If you want to activate a module, it is not necessary to reboot. The CDev will automatically check for new modules and newly activated modules when it is opened. So, to activate a previously inactive module, just remove the check from the "Info" box in the Finder and open the CDev. This only applies when making a module active, not when making it inactive... once a module has been loaded, it's there to stay. 2e. incompatibilities [ 1998/06/06: see the README file on the disk for a more accurate list ] Because of the way HardPressed works, there are some programs with which it will not get along. Here they are, from worst to least. - ProSel-16. As of v8.81, ProSel did a lot of low-level disk I/O, bypassing HardPressed. This is only a factor in some areas, notably the utilities. It would be wise to change the mode to Inactive when using ProSel's utilities, backup/restore, volume repair, and volume optimization menus. The author has used ProSel-16 as a program launcher throughout the development of HardPressed, and has had no problems with that part. However, it is *VERY* important that you do not try to copy compressed files with ProSel's utilities while HardPressed is active. - SoundStudio (freeware digitized sound utility). When loading sounds, this program uses the file's block size to compute the sound's length. Since HardPressed reduces the number of blocks that the file requires on disk, SoundStudio thinks the files are smaller than they really are, and only loads the first part of the file. - GNOBug (debug INIT for the GNO environment). "gnobug" is used to hook GNO in above GSBug, to make _osbrk work right. However, hooking GNO above HardPressed can cause problems. The correct order is HardPressed first, gnobug second, and GSBug third. [this may not be the case anymore... Jawaid?] - Old ORCA shells. Nothing serious here. The "CAT" command will show the files as HardPressed files (LBR), but all commands and utilities will function without any problems. 2f. Questions & Answers Q. Sometimes when I get Icon Info from the Finder (OA-I), it expands the file. Why? A. Some files have version information in the resource fork. For example, getting Icon Info on the HardPressed CDev will display the full name of the program and a copyright notice. To get at this information, the resource fork must be expanded first. The data fork is not examined. Q. I copied the "All: Adapative Huffman" profile out of the Goodies folder, but I can't seem to compress anything with it. A. The profiles tell HP which modules to use. If the needed modules aren't loaded, HP can't use them. By default, the Adaptive Huffman and Differential modules are marked as "inactive" in the Finder. You can enable them with the Finder's Icon Info feature (click on the module in the System/System.Setup/HP.Modules folder, hit OA-I, and click on the "inactive" box so that the box is empty). Once you have activated the module, it will be available every time you boot GS/OS. HardPressed will load it when you reboot or reopen the CDev. Q. When compressing a file with the Finder extension, why does the little document icon empty and then fill up? A. You have verify mode enabled. It empties during compression, and fills during the verify. When expanding, it just fills. III. How HardPressed Works [ 1998/06/06: some is explained better in files in the "Goodies" folder ] This chapter goes into greater technical detail than the previous. You don't need to read this to use HardPressed, but you will pick up some interesting tidbits even if you just skim it. One item which should be stressed is that HardPressed is as safe as GS/OS in theory, but may be less so in practice. This means that, if GS/OS says "you are safe", you are still safe. If it says, "you risk loss of data", then your risk is slightly higher. For example, suppose you have a power failure. GS/OS caches parts of files, so if you lose power while a file is open, you may lose data. HardPressed has a larger file cache, so the chance that your data will be in memory instead of on disk when the power goes down is higher. Thus, HardPressed does not create a new opportunity for data loss, but instead merely increases the odds of being caught in a bad situtation. (Of course, you can avoid this particular problem by setting the cache size to "Off".) 2a. file interface HardPressed identifies its data files by file and auxtype. (Actually, there are three auxtypes, indicating that the data fork, resource fork, or both are compressed.) This is one of several possible methods, and is the best for the //gs. If you have already installed and run HardPressed, you will have noticed that the files appeared to retain their original type. This is because HardPressed intercepts the GS/OS calls which get the files' types, and replaces the types with the original ones. The original filetype, auxtype, and EOF (end of file - the length of the file in bytes) are stored in the appropriate fork of the file itself. This means that the file must be opened and read whenever a program (such as the Finder) tries to access any of the above. At this point you might be thinking, "yuck, there must be a better way." Don't we wish. The main roadblock is the file EOF, which must reflect the ORIGINAL size, NOT the compressed size. Suppose we have a 10K file, which compressed down to 5K. Your Favorite Word Processor wants to edit the file, so it gets the file's length. If it sees "5K", it will open the file, read 5K, and stop. However, HardPressed will have uncompressed the file to its full 10K size, so you would only get half of the file. Thus, we need to tell FWP that the file's size is really 10K, or we may appear to lose data. This can still be a problem with programs which read the disk directly instead of making GS/OS file calls (ProSel-16 as of v8.81 did this). Because they read it directly, HardPressed doesn't get a chance to substitute the true file size, and the situation seen above happens. It is important to remember where the filetype is being stored when changing the access permissions on a file. If a file is read-disabled, HardPressed will have great difficulty getting the filetype. If it is write-disabled (i.e. "locked"), changing the type will be difficult. I say "difficult" and not "impossible" because if the file is locked, HardPressed will change the file's access bits, read or write the necessary information, and then reset them afterward. In general it is a very bad idea to disable read access, because accessing the file will be impossible. (An extreme example: if you open a locked file with read-only access, change the filetype, and then close the file, HardPressed will close the file, change the access bits, reopen the file with read/write access, change the filetype, close the file, relock the file, and reset the file's modification date to its previous value. How's THAT for service?) As I just hinted at, it IS possible to reliably change the filetype on an open file. Normally you can't write anything to a file which is already open. In this case, however, the file will have been opened by HardPressed, which can just change its internal data structures and save the information when it goes to close the file. This will not work if you deliberately try to defeat it (by turning HardPressed off from the CDev, opening the file, and then turning it back on again), and it may not always work on AppleShare volumes, where a file could have been opened by someone elsewhere on the network. HardPressed does everything it can to avoid these problems, but there are some situations which are simply impossible to resolve (for example, the file/auxtype/EOF information could be corrupted). In such cases, queries on the file information will return HardPressed's filetype ($e0 - LBR) and the actual EOF, and attempts to access the data inside file will fail with error $4A (invalid version). 2b. hard at work Small bit of background for those unfamiliar with GS/OS... The "Open" call prepares a file for reading or writing. You give it a pathname, and it returns a "reference number" or "refNum". All further accesses are done with the refNum instead of the pathname. The "Read" call reads data from a file, and the "Write" call writes data. The "Close" call tells the system that you are finished with the file. NOTE: These are _not_ the same things as the Open and Close items found in the "File" menu of desktop applications. Choosing "Open..." from the menu bar may cause application programs to invoke several GS/OS Open and Close calls. (Remember, you don't NEED to know all this stuff; we just put it here in case you got curious.) When an "Open" call is issued, HardPressed opens the file and checks the filetype and auxtype to see if the data is compressed. If so, the first few bytes of the file are read to find out how big it really is and how it was compressed. If it doesn't know how to deal with the file (perhaps a needed compression module was marked inactive), it closes it and returns an "invalid version" error. When the first Read or Write call is issued, the entire file is expanded (more accurately, the entire FORK is expanded). Depending on a number of factors, the uncompressed data will be stored either in the HardPressed file cache or in a temporary file on disk. Future Reads and Writes are made to the file in memory or the file on disk. Note that, even if the temporary file was too big to store in the cache, HardPressed will try to cache any Write calls which fit. When the last Close call is issued (you can issue more than one Open on a file, so you can have an equal number of Closes), HardPressed checks to see if the file's contents or filetype have changed. If the contents have changed, the file is recompressed from the data in the cache or in the temp file. If only the type has changed, then just the header will be rewritten. If the file is smaller than 512 bytes uncompressed, it will always be stored without compression. This is because the typical disk block size is 512 bytes, and the file will occupy one full disk block whether it's 5 bytes or 511. The Finder extension silently ignores tiny files. When an Open call is issued on an UNcompressed file, HardPressed checks the mark status of the folder it's in and the file's access bits. If it's writeable and in a "compress when changed" folder, HardPressed will figure out how to compress it (explained next) and add it to the list of files it's handling. From there it works like a file which was already compressed: if it gets changed, it gets compressed; if not, nothing is written. 2c. HowPressed [this might belong in the basic chapter, actually] As mentioned earlier, HardPressed can compress your files in several different ways. These are summarized in the table below. [put the "cdref" file in a table, I guess] A significant feature of all the compression modules is that they can be chained together. For example, you could compute a CRC, pack the file with RLE, and then pack it again with LZSS. Future compressors - and other modules, including stuff like data encryption or error correcting codes - can simply be chained on. The output is "piped" directly between them, so you don't have to wait for disk access between each stage. Besides being able to add fancy stuff like CRCs or data encryption to the compression, this enables you to compress files in ways which would have been difficult otherwise. For example, digitized sound files don't compress very well with standard algorithms. However, if you perform a "differential" encoding pass followed by adaptive Huffman, you can get fairly good results. Another example is "sparse" files. Sparse files are large files (perhaps several megabytes) which are almost entirely filled with zeros. Typical compression algorithms will have spend a lot of time compressing all the zeros before they can compress the data. However, the RLE (Run Length Encoding) algorithm was designed to deal with large stretches of zero bytes, so by chaining RLE and LZW together, you can compress the zeros quickly but still get the benefits of LZW compression. The way that compression modules are chosen is through the compression profiles you select from the CDev. Each profile contains a list of "categories", which group files by filetype and auxtype. There's a category for applications, one for graphics, one for word processing files, and so on. Each category has up to eight modules associated with it. When a file is Opened, HardPressed determines which category the file belongs in, and then looks up the list of modules for that category. If there aren't any, the file will not be compressed. This applies to files which are already compressed as well: if you recompress a file, it will use whatever the system says to do now, NOT what was used when the file was compressed. There's a couple of good reasons for doing it that way. First, it allows you to do some neat things, which will be explained next. Second, it isn't always possible to determine what kind of compression was used previously. For example, some applications will delete the old data file before creating a new one. When the old file gets deleted, all of the information about how it was compressed goes with it. Other applications save the data to a temporary file, and then when the save is completed they replace the original file with the temporary one, so again there's no way to retrieve the previous information. There are several profiles provided for you. "Standard" gives you good compression at moderate speed. "Faster" gives you less compression, but uses faster modules and compresses fewer kinds of files. "Smaller" gives you better compression at the cost of speed. There are also some modules which compress absolutely everything (including ProDOS 8 stuff!!) with one particular module. These are useful if you want to compress a lot of data in one particular way. If you examine the list of compression modules carefully, you'll notice that LZSS is the best at packing files (marginally edging out LZW), and unpacks quickly (blowing away LZW). However, it has truly abysmal performance when compressing. This leads to an interesting disk compression strategy. Basically, you choose the "only LZSS" profile from the CDev, select several folders worth of stuff from the Finder, select "Compress" from the Extras menu, and go out to lunch (or to sleep if you have a large hard drive). When you come back, all of your files will be tightly compressed, and will extract quickly. You should then switch back to one of the standard profiles. Thereafter, when you change a file or create a new one, it will be compressed with the method in the profile, NOT LZSS. This allows you to get the benefits of tight compression with no reduction in speed in daily operations. 2c.1. help, I've expanded and I CAN'T GET UP If a file fails to compress (i.e. the file is larger after compression than it was before), HardPressed will store the file without compression. This can happen for certain kinds of files, such as digitized sound or files which are already compressed. One interesting feature is that, if you have included some modules which are NOT compressors (such as data encryption or error correction), they will not be included in the comparison, and they will not be removed should the file expand. Suppose you have some super duper encryption scheme which expands the file by 50%. You compress it by 10% with RLE, and then encrypt it. The resulting file is larger than the original, but because the COMPRESSION modules made it smaller, it will be kept. On the other hand, suppose RLE fails and increases the size by 10%. In that case, HardPressed will re-encode it with ONLY the encryption module. At present, all of the modules are defined as compressors, including the CRC module (this was done so that you wouldn't end up with a bunch of stray files with just CRCs on them). Assigning compression methods into categories is done by the stand-alone application [which hasn't been written yet. Oh bother.] 2d. the big off If you've been reading along, you now have enough background to understand what the "off" switch in the CDev and CDA REALLY does. There are two places in which the switch is examined. First, when handling file information calls. When HardPressed is "off", all such calls are passed directly to the system. This means you will get the original filetype, auxtype, and EOF marker, and changes to the file's type will affect its "true" type. If you accidentally mangle a file's type, you can have it reset automatically from the stand-alone application. The second activity it affects is when deciding whether or not to compress a file. If the switch is off, all opens are passed directly to the system. The switch does NOT affect any other calls, so Read, Write, and Close will continue to behave as before. This means that files that were open BEFORE you threw the switch will continue to be handled correctly. However, if the switch is off when the Close call is issued, they will be stored without compression. The "Expand only" setting has a similar effect, except that it doesn't affect the file information calls. It just changes the policy decisions in Open and Close so that uncompressed files stay that way and compressed files will not be recompressed. 2e. welcome to moduland The compression modules are actually "filters"; any algorithm which can operate on a continuous stream of data can be built into a module. This means that data compression is only one possible use for them. Data encryption, error detection (like the CRC module included), error correction, and addition of extra file information (like a modification time in seconds or a comment field) are all possible. A "HardPressed module author's kit" will be made available on the major online services. It will explain in detail how the modules work, and will include some sample source code. In addition, lesson 12 of "Hacking Data Compression", an online course in the A2Pro section of GEnie, will be all about writing your own modules. One recurring daydream is to write a module which will encrypt a file in a way that only works on your system. The modules could use the message facilities in System 6 to communicate with a "key server" running on your system as an invisible INIT. So long as the INIT is there, your files are accessible. If not, they aren't. This would allow you to have encrypted files accessible without having to type a password all the time (useful in a networked environment where you can guarantee the physical safety of your local machine), or at least have the password program be indepdendent of the module itself. It could also be used as a form of copy protection (boo, hiss!) Well, back to Earth. The compression modules are stored along with the profiles in a folder called "HP.Modules" under */System/System.Setup. (Well, it looks for the folder in the same directory that the HardPressed INIT is in, so it should work correctly in an AppleShare environment.) If it doesn't find the folder, it will use a standard profile that is built in to it, but won't actually be able to compress or uncompress anything. Adding new or updated modules to your system is easy enough: just copy them into the HP.Modules folder and restart your system. In addition, HardPressed will scan for new modules and load them whenever the CDev is opened. Experience has shown that automatically updating to a new version of a file by checking the modification dates is unreliable, so this is not attempted. Compression is, by nature, a CPU-intensive activity. HardPressed will do a lot better with an accelerator board, especially one with 32K or more of cache memory (the compression code is small, but the data structures and output buffer are often large). 2f. configure me a volume The volume configuration is kept in a file called "HP.Vol.Prefs", stored in the volume directory in an invisible file. It contains a general policy for the volume, and a list of marked folders. Each entry affects all folders beneath it, so they are stored in deepest-to-shallowest order, with the entry for the volume directory being considered last. If this file is removed, the entire volume will be considered as "Expand only". However, it may not take effect immediately... To increase speed, up to 8 volume configuration files are kept in memory at a time. If you have a lot of volumes online, all of the available slots will eventually fill up. New ones will replace old ones on an LRU (Least Recently Used) basis. That is, the configuration information that hasn't been accessed for the longest time will be erased from memory, and the new one will be loaded into its slot. This does NOT affect the files on disk, which are never modified by anything except the CDev. These actions will be completely invisible; the delay caused by reading them is almost zero. Some actions will cause the copy in memory to be marked as invalid. Since there is no way to determine if a user has ejected a disk labeled ":foo" and inserted a different disk also labeled ":foo", it is necessary to invalidate the entry for ":foo" when the disk is switched. All of the entries will be wiped if a new file is written from the CDev's folder preferences screen (i.e., you have to make a change to something; just opening up a disk and looking at the contents isn't sufficient). 2g. temporary insanity From the CDev, you can choose to store temporary files in a single directory or in the same directory as the file it's paired with. A discussion about which is best can be found with the description of the CDev earlier. If HardPressed is unable to create the temp file in the current directory, then it will try to create it in the configured temp directory. If that fails, the operation will abort. Internally, ALL files have a temporary file associated with them. This includes files which are uncompressed but are candidates for compression. However, not all temporary files reside on disk. For example, when you open an uncompressed file which is eligible for compression, HardPressed makes a note that the temp file does not yet exist, and makes no attempt to create one. Then, if the file is to be compressed, a temp file is opened to store the compressed data while HardPressed is working. This makes access to files which aren't candidates for compression as fast as for those which are. When a compressed file is opened, no temp file is created on disk until it is actually needed. Since it's possible for a temp file to reside entirely in the file cache, a disk file might never be created at all. This makes many common operations MUCH faster. For the case of compressing a file for the first time, the data is read from the uncompressed file and written (after being compressed) to the temporary file. When finished, the contents of the temp file are copied over. To make some common operations (such as file copying) faster, HardPressed checks to see if the original file is empty. If so, all writes are done in the temporary file, and the compressed data is written directly to the original file. This avoids the extra copy. It may appear that all of this copying around creates an extra risk of losing data, because the only copy of the original file is getting overwritten by the compressed data. However, this is not the case. If something fails in the middle of the copy, HardPressed will restore the original file from the compressed copy. If THAT fails (because of I/O errors or other unexpected occurrences), then you are no worse off than you would have been without HardPressed, because it's doing the exact same thing (i.e. writing uncompressed data) that would have been done if HardPressed weren't loaded. The "verify" option (discussed later) happens BEFORE the uncompressed copy is altered, so there is no risk of errors in HardPressed producing an unusable compressed copy. Temporary files will have names like "HP0005", and will have file and aux types which match the original files. They are automatically deleted when they are no longer needed, so you won't usually see them unless the system crashes. 2h. trust, but verify Our first reaction to queries about a verify option was, if we thought it was necessary, we wouldn't ship the product. However, you may not have the same faith in HardPressed as we do, so we provide a way to verify that the compressed copy is good before removing the uncompressed copy. When you enable verification, HardPressed does the usual set of stuff when the file is closed, but does an extra step after compression has successfully completed. It turns around and expands the file again, verifying that the data that it expands matches the original file exactly. The comparison is done by computing a CRC (Cyclic Redundancy Check) on the data before compression and after expansion. When verifying, the file is expanded to memory, but not stored on disk. This makes the verify pass fairly fast. [if you've got the border color change stuff going, you will notice that it stays red for slightly longer because of the CRC, but the pink phase is very fast.] 2i. no news is good news Normally, HardPressed is completely invisible. However, if it is unable to complete an operation, it will put a message on your screen. If you are using a desktop application, you will get a standard alert dialog. If not, you will get a message on the text screen. This is similar to what GS/OS does. There are four classes of messages, Internal Errors, Sanity Failures, Warnings, and Info messages. Internal Errors happen when something unexpected failed. Being the responsible sort, HardPressed blames itself for the mishap, reports it, and aborts the current operation. Sanity Failures happen when HardPressed checks itself and finds that something is amiss (possibly due to somebody else's buggy program munging random bytes). There are dozens of "sanity checks" sprinkled throughout HardPressed; these help to ensure the integrity of your data [and make it easier to find bugs!] Warnings are messages about mundane errors, like running out of memory or disk space. Most common errors are just passed straight back to the application, but some errors (like running out of space in the temp folder) require your more immediate attention. Info messages are just general informative messages. [at present there aren't any.] Sometimes it may appear that HardPressed will go insane and continually spit error dialogs at you. If it does, it's because it was unable to complete something you asked for earlier, and it's afraid to abort it because of the risk of losing data. For example, if you tried to save a file but ran out of space in your temporary folder (a bad thing), it will hold that file open until you free up space. This can be annoying, but it does allow you the chance to stop what you're doing and free up space without the risk of losing valuable data. IV. Programmers Only The following GS/OS calls are intercepted: $03 OSShutdown $05 SetFileInfo $06 GetFileInfo $10 Open $11 NewLine $12 Read $13 Write $14 Close $15 Flush $16 SetMark $17 GetMark $18 SetEOF $19 GetEOF $1c GetDirEntry $24 Format $25 EraseDisk $29 Quit $38 GetRefNum $39 GetRefInfo The following may be handled in future releases: $01 Create $02 Destroy $04 ChangePath Class 0 calls are translated into their class 1 equivalents before processing. No information is gained or lost by this procedure. The class 0 call GET_LAST_DEV is usually unreliable under GS/OS without taking extreme cautions, and may be even more so with HardPressed active. (Consult the GS/OS reference.) Most of the handlers will make one or more GS/OS calls. NewLine is the only one which NEVER does. To maintain GS/OS semantics, Null ($0d) is called whenever a call is handled entirely by HardPressed. The interface code raises an internal busy flag and the system busy flag (via incbusy and decbusy), which should prevent problems with task switchers like GNO. Calls which are not handled by HardPressed (including those not listed above as well as stuff like a Read on a refNum we don't handle) are executed AFTER both of the busy flags have been dropped. This allows Quit, OSShutdown, and Read on .console to function correctly. The amount of time required per call should be unnoticeable for most applications, but is optimized toward large transfers. Applications doing character-by-character I/O will not fare well, but if they're using character I/O they're probably slow to begin with. (Note that this does NOT include C programs using the standard I/O routines like getc(), putc(), printf(), and so on; those are buffered to 1K or 4K.) One glaring exception is Orca shell v2.0, which issues a GS/OS Write call for every character (this is why "cat" is so slow). To reduce the overhead, the interface has some special code which detects multiple consecutive writes to the same file, and circumvents the usual processing. As a result, the reduction in speed is unnoticeable (176 cycles per character). The transition to ProDOS 8 is a bit tricky. HardPressed has 4 pages of direct page space which are permanently resident (stack/DP for HardPressed, stack/DP for the compression modules). Because ProDOS 8 applications have all of bank 0 available to them, this will cause a $0201 error if they are not disposed of. When the GS/OS notification mechanism warns of an impending GS/OS to ProDOS 8 transition, HardPressed goes into "swap mode", and will swap all 4 pages in and out on every GS/OS call. It's ugly, but it works. When the P8 to GS/OS notification arrives, HardPressed reinstalls itself into the GS/OS vector and switches out of swap mode. This means that HardPressed will be active until the very last second, so any files accessed on the way in or out will be handled correctly. (And yes, the notifications arrive in the order in which they were requested, so HardPressed should be the first one back on.) Communication between the various components is done with the AcceptRequest and SendRequest calls added in System 6.0. Most of the requests would be dangerous to execute if you don't know all the consequences, but the ones which turn HardPressed on and off are pretty straightforward. Note that no error checking is done. idstr dw 'WestCode~HardPressed~Main~' dMping equ $8000 ;does nothing dMsetStatus equ $8001 ;set status (on/off/decode) dMgetStatus equ $8002 ;get status dVpolDecode equ $0001 ;decode-only dVpolOn equ $0002 ;system on dVpolOff equ $0003 ;system off fGverify equ $4000 ;verify on/off fGoneTemp equ $2000 ;use a single temp dir fGmouseTrix equ $1000 ;show progress with cursor For setStatus, send the status as the low word of dataIn. For getStatus, the status will be at dataOut+2 (right after the count word). DO NOT send values other than those above, and DO NOT send messages other than those above. For example, if you wanted to set the status to "decode-only" mode, you would issue a getStatus call, AND the status with #$ff00, OR it with #dVpolDecode, and then issue a setStatus call. I reserve the right to add new flags at any time and without notice; BE SURE you aren't turning on or off flags you don't know about. All I guarantee is that the global_flags are in the hi byte and the global_mode is in the low byte. Yes, there are other messages. DO NOT use them. I mean it. The source code for the CDA can be found in the "Goodies" folder; this illustrates the use of the above calls. The file/aux type assignments for HardPressed files are: Filetype (generic name) | Aux type | Kind as Finder(tm) will report -------------------------------------------------------------------- $5A (Configuration file) | $804B | HardPressed volume preferences $5A (Configuration file) | $804C | HardPressed global preferences $5A (Configuration file) | $804D | HardPressed profile $BC (Generic load file) | $400F | HardPressed module (also $C00F) $E0 (Archival/library) | $800D | HardPressed compressed data $E0 (Archival/library) | $800E | HardPressed compressed data $E0 (Archival/library) | $800F | HardPressed compressed data $800d is for a compressed data fork / uncompressed resource fork; $800e is for a compressed resource fork / uncompressed data fork; $800f is for both forks compressed. [ perhaps add the "hazards" file here? ] Appendix A : Standard Profile Contents [ list of categories ] [ matching of file/auxtype combinations with categories ]