Name: Anonymous 2009-01-20 16:36
To become a truly EXPERT PROGRAMMER, you must learn and appreciate the art of requirements and specifications (which, by meta requirement, change within days of product delivery).
This example case will be designing the ultimate *chan board monitor/web spider.
Requirements:
ID#01: Toggle monitoring a single thread
ID#02: Monitor a board for new threads
ID#03: Minimize bandwidth as much as possible
ID#04: Allow browsing of the downloaded content immediately
ID#05: Allow viewing of the board separate from the monitor (but should not duplicate downloads)
ID#06: Should be able to pickup where it left off after restart or crash
ID#07: The number of connections should be configurable
ID#08: Cross-platform
ID#09: Provide automatic criteria for what threads to start monitoring
ID#10: Automatically post replies to threads/posts that meet certain criteria
ID#11: Mass spamming or file uploading
ID#12: Options to download thread text, thumbnails, and/or full images
ID#13: Reduce storage/data redundancy
ID#14: Option to set how often to check for changes
ID#15: Keep memory use to a minimum
Some implementation details:
ID#01: Toggle monitoring a single thread
* When full board monitoring is enabled, this can tell it to ignore one thread
* When full board monitoring is enabled, this could add the thread before the board monitor encounters it
ID#02: Monitor a board for new or changed threads
* While monitoring the entire board, there's no need to download individual threads unless they are seen to have changed (post added, post deleted, image deleted, etc.)
ID#03: Minimize bandwidth as much as possible
* Always Use Last-Modified HTTP tag
* Use .gz transfer as much as possible
* Keep database of all downloaded images: file type, size, dimensions, md5
* Check the md5, file type, and size before downloading images
* Use identical software the *chan board uses to generate thumbnails so they can be generated automatically without downloading
* Alternatively use the thumbnail to do a fuzzy comparison (can only really determine if we don't already have the file for sure)
ID#06: Should be able to pickup where it left off after restart or crash
Persistent storage
* details about all images ever downloaded
* queue of pages/files to download
* queue of downloaded data to process
* thread details: number of posts, number of images, sticky, last post id, HTTP Last-Modified, marked for delete, banned count
* post details:
ID#13: Reduce storage/data redundancy
Use pixel perfect comparison, and fuzzy comparison of images to determine which are identical or similar.
ID#15: Keep memory use to a minimum
Keep most data in persistent storage.
Given the above, there are 2 possible development platforms:
* A Firefox extension would be ideal
- It could use pages you've browsed as part of its downloading and processing
- You can also immediately browse downloaded and processed threads
- Posting to the board is still available, and can even track the images you upload so it doesn't download them again but instead copies the local copy
* Alternative platform
- Create an HTTP daemon that serves up all of its content
- This could also be used by multiple people, either in an intranet, or even the internet
- Essentially creating a reformatted and optimized mirror of the *chan board
- Posting to the board may not be available
Areas for further specification:
* Flush out all the requirements
* Add use cases
* Specify various necessary modules (persistent storage, connection handling, html parsing, etc)
This example case will be designing the ultimate *chan board monitor/web spider.
Requirements:
ID#01: Toggle monitoring a single thread
ID#02: Monitor a board for new threads
ID#03: Minimize bandwidth as much as possible
ID#04: Allow browsing of the downloaded content immediately
ID#05: Allow viewing of the board separate from the monitor (but should not duplicate downloads)
ID#06: Should be able to pickup where it left off after restart or crash
ID#07: The number of connections should be configurable
ID#08: Cross-platform
ID#09: Provide automatic criteria for what threads to start monitoring
ID#10: Automatically post replies to threads/posts that meet certain criteria
ID#11: Mass spamming or file uploading
ID#12: Options to download thread text, thumbnails, and/or full images
ID#13: Reduce storage/data redundancy
ID#14: Option to set how often to check for changes
ID#15: Keep memory use to a minimum
Some implementation details:
ID#01: Toggle monitoring a single thread
* When full board monitoring is enabled, this can tell it to ignore one thread
* When full board monitoring is enabled, this could add the thread before the board monitor encounters it
ID#02: Monitor a board for new or changed threads
* While monitoring the entire board, there's no need to download individual threads unless they are seen to have changed (post added, post deleted, image deleted, etc.)
ID#03: Minimize bandwidth as much as possible
* Always Use Last-Modified HTTP tag
* Use .gz transfer as much as possible
* Keep database of all downloaded images: file type, size, dimensions, md5
* Check the md5, file type, and size before downloading images
* Use identical software the *chan board uses to generate thumbnails so they can be generated automatically without downloading
* Alternatively use the thumbnail to do a fuzzy comparison (can only really determine if we don't already have the file for sure)
ID#06: Should be able to pickup where it left off after restart or crash
Persistent storage
* details about all images ever downloaded
* queue of pages/files to download
* queue of downloaded data to process
* thread details: number of posts, number of images, sticky, last post id, HTTP Last-Modified, marked for delete, banned count
* post details:
ID#13: Reduce storage/data redundancy
Use pixel perfect comparison, and fuzzy comparison of images to determine which are identical or similar.
ID#15: Keep memory use to a minimum
Keep most data in persistent storage.
Given the above, there are 2 possible development platforms:
* A Firefox extension would be ideal
- It could use pages you've browsed as part of its downloading and processing
- You can also immediately browse downloaded and processed threads
- Posting to the board is still available, and can even track the images you upload so it doesn't download them again but instead copies the local copy
* Alternative platform
- Create an HTTP daemon that serves up all of its content
- This could also be used by multiple people, either in an intranet, or even the internet
- Essentially creating a reformatted and optimized mirror of the *chan board
- Posting to the board may not be available
Areas for further specification:
* Flush out all the requirements
* Add use cases
* Specify various necessary modules (persistent storage, connection handling, html parsing, etc)