Verification and Validation
Michael Wales - NeedMoreHP@yahoo.com
EEL 5874 - Expert Systems and Knowledge Engineering
Version History:
Document | Date | Description |
Verification and Validation V1.0 | April 15, 2002 | Initial Document |
Verification and Validation V2.0 | April 17, 2002 | Addded the test results |
Verification and Validation V3.0 | April 18, 2002 | Shaun Murphy (the expert) evaluated the test results |
Table of Contents
Verification Criteria:
The verification for the required ComputerAdvice system will focus on the application following the correct procedures, without any errors. The verification phase is not concerned about the correctness of the output. The following table lists the verification criteria for the system.
Required System - Command prompt system
For the required command prompt system, all of the above verification requirements need to be completed in full. In each and every test case for the application, all of the above requirements should be satisfied. If any requirement is not met during testing, the system will be examined and corrected, and then tested in full again.
Optional System - Web based system
Since this part of the application is optional, not all of these verification criteria need to be met. If any of the optional system verification criteria fails during testing, an effort will be made to correct the problem. If the time required to fix the system will be large enough to affect the development and testing of the required system, the optional system will remain as-is.
Validation Criteria:
The validation phase concentrates on examining the output of the system for correctness. The system will be investigated to make sure that the output is correct, but also that the reasoning is correct also. Validation also makes sure that then end product is what the end user expected.
The validation is the same for both the required system (command prompt based) and the optional system (web based). The systems only differ in how the user interacts with the system, the knowledge engine is identical for each system. The validation will likely occur with the command prompt system simply because it is faster than the web based system.
The validation methodology used will be a hybrid of informal validation and validation by testing. Informal validation is where the developer of the system would meet with the expert and informally test the system. Validation by testing is where test cases are used to make sure that the system agrees with the expert's results. The will use a hybrid of these two processes because the problem my system tries to solve never has a real definite answer.
The validation process will be as follows:
The ComputerAdvice system shall get ratings of strongly agree, agree, and acceptable. If the expert finds the ComputerAdvice recommendation unacceptable, the knowledge will be tuned to get the desired results, and then retested. If the expert and ComputerAdvice disagree, the reasoning used to get there will be looked at for problems. If there appears to be good reason for the ComputerAdvice recommendation, it will be accepted.
Test Cases:
Case 1: Gaming and Home Office computer (Medium price point)
Verification Criteria | Passed |
The system shall be free from compilation errors (and warnings), runtime errors, and shall not throw any unhandled exceptions. | Passed |
Given a sufficient budget, the system shall always be able to create a functional computer system. | Passed |
The application shall prompt the user for all of the possible common computer applications. | Passed |
The system shall prompt the user to enter a number for the computer system budget. | Passed |
The system shall print out a computer system build sheet. | Passed |
R1 -> The system shall output the recommended system as a plain text table with each component's description and price. | Passed |
R2 -> The system shall output the recommended systems estimated total price. | Passed |
R5 -> The system shall ask the user the importance of performance of each of the common computer applications. | Passed |
R6 -> The system shall ask the user what they can afford to spend on the new computer system. | Passed |
Expert's Price Estimate | $800 |
Expert's Build Sheet | Duron 1.3GHz CPU, 256 MB DDR-RAM, 40GB HD, Gigabyte DDR motherboard, CDROM, 300W Case, GeForce3 Ti200 |
Computer Advice Inputs | Use gaming applications Use home office applications $800 budget |
Computer Advice Build Sheet | Level Price Description ------------------------------------------------------- 1 $50 Duron 1.1 GHz 0 $44 Enlight Medium ATX Tower w/ 300W PS 2 $105 GeForce4 MX440 0 $20 Standard 56k Winmodem / 100BTX NIC 1 $64 Creative Labs Audigy 5.1 0 $65 20GB WD 7200 RPM Hard Drive 0 $25 52x CD-ROM Drive 0 $58 16x CD-RW Drive 0 $20 Labtec / Altec Lansing 2pc Speakers 1 $80 HP Deskjet 845C (8ppm, 600x600) 0 $79 HP ScanJet Flatbed 220C (600dpi, 42bit color) 1 $73 256MB PC2100 DDR 266MHz 1 $88 GigaByte 7VKML KM266 (DDR, Sound, Video, NIC) 0 $11 Microsoft Intelli Mouse PS2 0 $9 Mitsumi 104-key PS2 Keyboard 0 $9 Mitsumi 1.44MB 3.5in Floppy Drive ------------------------------------------------------- Total: $800 |
Computer Advice Rating | Strongly Agree |
Expert Comments | Good match for application. |
Case 2: Video editing computer (High price point)
Verification Criteria | Passed |
The system shall be free from compilation errors (and warnings), runtime errors, and shall not throw any unhandled exceptions. | Passed |
Given a sufficient budget, the system shall always be able to create a functional computer system. | Passed |
The application shall prompt the user for all of the possible common computer applications. | Passed |
The system shall prompt the user to enter a number for the computer system budget. | Passed |
The system shall print out a computer system build sheet. | Passed |
R1 -> The system shall output the recommended system as a plain text table with each component's description and price. | Passed |
R2 -> The system shall output the recommended systems estimated total price. | Passed |
R5 -> The system shall ask the user the importance of performance of each of the common computer applications. | Passed |
R6 -> The system shall ask the user what they can afford to spend on the new computer system. | Passed |
Expert's Price Estimate | $1100 |
Expert's Build Sheet | Athlon XP 2100+ CPU, 1GB DDR-RAM, 120GB HD, DVD-ROM, DVD-RW, 340W Case, Gigabyte DDR Motherboard, AIW Radeon7500 |
Computer Advice Inputs | Use video-editing applications $1100 budget |
Computer Advice Build Sheet | Level Price Description ------------------------------------------------------- 4 $107 Athlon XP 1800+ 2 $61 Enlight Medium ATX Tower w/ 340W PS 0 $20 Standard 56k Winmodem / 100BTX NIC 3 $66 Creative Labs Audigy 5.1 5 $173 120GB WD 7200 RPM Hard Drive 1 $48 16x DVD-ROM Drive 4 $62 16x CD-RW Drive 2 $60 Creative Labs 4.1 Speakers (6W x 4, 17W Sub) 2 $111 All-In-Wonder 128 Pro 16MB PCI 4 $214 256MB + 512MB PC2100 DDR 266MHz (768MB Total) 4 $148 Asus A7N266 nForce (DDR, Sound, Video) 0 $11 Microsoft Intelli Mouse PS2 0 $9 Mitsumi 104-key PS2 Keyboard 0 $9 Mitsumi 1.44MB 3.5in Floppy Drive ------------------------------------------------------- Total: $1099 |
Computer Advice Rating | Agree |
Expert Comments | Would've liked to see the DVD-RW in this system. |
Case 3: Bare bones home office system (Low price point)
Verification Criteria | Passed |
The system shall be free from compilation errors (and warnings), runtime errors, and shall not throw any unhandled exceptions. | Passed |
Given a sufficient budget, the system shall always be able to create a functional computer system. | Passed |
The application shall prompt the user for all of the possible common computer applications. | Passed |
The system shall prompt the user to enter a number for the computer system budget. | Passed |
The system shall print out a computer system build sheet. | Passed |
R1 -> The system shall output the recommended system as a plain text table with each component's description and price. | Passed |
R2 -> The system shall output the recommended systems estimated total price. | Passed |
R5 -> The system shall ask the user the importance of performance of each of the common computer applications. | Passed |
R6 -> The system shall ask the user what they can afford to spend on the new computer system. | Passed |
Expert's Price Estimate | $1100 |
Expert's Build Sheet | Athlon XP 2100+ CPU, 1GB DDR-RAM, 120GB HD, DVD-ROM, DVD-RW, 340W Case, Gigabyte DDR Motherboard, AIW Radeon7500 |
Computer Advice Inputs | Use video-editing applications $1100 budget |
Computer Advice Build Sheet | Level Price Description ------------------------------------------------------- 4 $107 Athlon XP 1800+ 2 $61 Enlight Medium ATX Tower w/ 340W PS 0 $20 Standard 56k Winmodem / 100BTX NIC 3 $66 Creative Labs Audigy 5.1 5 $173 120GB WD 7200 RPM Hard Drive 1 $48 16x DVD-ROM Drive 4 $62 16x CD-RW Drive 2 $60 Creative Labs 4.1 Speakers (6W x 4, 17W Sub) 2 $111 All-In-Wonder 128 Pro 16MB PCI 4 $214 256MB + 512MB PC2100 DDR 266MHz (768MB Total) 4 $148 Asus A7N266 nForce (DDR, Sound, Video) 0 $11 Microsoft Intelli Mouse PS2 0 $9 Mitsumi 104-key PS2 Keyboard 0 $9 Mitsumi 1.44MB 3.5in Floppy Drive ------------------------------------------------------- Total: $1099 |
Computer Advice Rating | Agree |
Expert Comments | This system might be more affordable with a lower levels on CPU and Hard drive selection. |
Case 4: Web-server computer (Medium price point)
Verification Criteria | Passed |
The system shall be free from compilation errors (and warnings), runtime errors, and shall not throw any unhandled exceptions. | Passed |
Given a sufficient budget, the system shall always be able to create a functional computer system. | Passed |
The application shall prompt the user for all of the possible common computer applications. | Passed |
The system shall prompt the user to enter a number for the computer system budget. | Passed |
The system shall print out a computer system build sheet. | Passed |
R1 -> The system shall output the recommended system as a plain text table with each component's description and price. | Passed |
R2 -> The system shall output the recommended systems estimated total price. | Passed |
R5 -> The system shall ask the user the importance of performance of each of the common computer applications. | Passed |
R6 -> The system shall ask the user what they can afford to spend on the new computer system. | Passed |
Expert's Price Estimate | $500 |
Expert's Build Sheet | Duron 950MHz CPU, 300W Case, 512MB PC133 RAM, 120MB HD, CDROM |
Computer Advice Inputs | Use webserver applications $500 budget |
Computer Advice Build Sheet | Level Price Description ------------------------------------------------------- 1 $50 Duron 1.1 GHz 2 $61 Enlight Medium ATX Tower w/ 340W PS 0 $48 GeForce2 MX200 0 $20 Standard 56k Winmodem / 100BTX NIC 2 $95 60GB Maxtor 7200 RPM Hard Drive 0 $25 52x CD-ROM Drive 1 $73 256MB PC2100 DDR 266MHz 1 $88 GigaByte 7VKML KM266 (DDR, Sound, Video, NIC) 0 $11 Microsoft Intelli Mouse PS2 0 $9 Mitsumi 104-key PS2 Keyboard 0 $9 Mitsumi 1.44MB 3.5in Floppy Drive ------------------------------------------------------- Total: $489 |
Computer Advice Rating | Agree. |
Expert Comments | A higher amount of PC133 ram (not DDR) is a prefered solution to this application. |
Case 5: Home Theatre computer (Medium price point)
Verification Criteria | Passed |
The system shall be free from compilation errors (and warnings), runtime errors, and shall not throw any unhandled exceptions. | Passed |
Given a sufficient budget, the system shall always be able to create a functional computer system. | Passed |
The application shall prompt the user for all of the possible common computer applications. | Passed |
The system shall prompt the user to enter a number for the computer system budget. | Passed |
The system shall print out a computer system build sheet. | Passed |
R1 -> The system shall output the recommended system as a plain text table with each component's description and price. | Passed |
R2 -> The system shall output the recommended systems estimated total price. | Passed |
R5 -> The system shall ask the user the importance of performance of each of the common computer applications. | Passed |
R6 -> The system shall ask the user what they can afford to spend on the new computer system. | Passed |
Expert's Price Estimate | $1100 |
Expert's Build Sheet | AthlonXP 1800+ CPU, 300W Case, 512MB DDR-RAM, GeForce3 Ti200, Sound Blaster Audigy Platinum, 60GB HD, DVD-ROM, Premium Surround Sound Speakers |
Computer Advice Inputs | Use multimedia server applications Audiophile system Budget $1100 |
Computer Advice Build Sheet | Level Price Description ------------------------------------------------------- 1 $50 Duron 1.1 GHz 2 $61 Enlight Medium ATX Tower w/ 340W PS 3 $156 GeForce3 Ti200 0 $20 Standard 56k Winmodem / 100BTX NIC 3 $66 Creative Labs Audigy 5.1 5 $173 120GB WD 7200 RPM Hard Drive 1 $48 16x DVD-ROM Drive 4 $62 16x CD-RW Drive 3 $100 Createive Labs 5.1 Speakers (6W x 5, 18W Sub) 4 $214 256MB + 512MB PC2100 DDR 266MHz (768MB Total) 2 $121 ABit NV7M nForce (DDR, Sound, Video, NIC) 0 $11 Microsoft Intelli Mouse PS2 0 $9 Mitsumi 104-key PS2 Keyboard 0 $9 Mitsumi 1.44MB 3.5in Floppy Drive ------------------------------------------------------- Total: $1100 |
Computer Advice Rating | Strongly Agree |
Expert Comments | Good match. |
Case 6: Graphics development computer (High price point)
Verification Criteria | Passed |
The system shall be free from compilation errors (and warnings), runtime errors, and shall not throw any unhandled exceptions. | Passed |
Given a sufficient budget, the system shall always be able to create a functional computer system. | Passed |
The application shall prompt the user for all of the possible common computer applications. | Passed |
The system shall prompt the user to enter a number for the computer system budget. | Passed |
The system shall print out a computer system build sheet. | Passed |
R1 -> The system shall output the recommended system as a plain text table with each component's description and price. | Passed |
R2 -> The system shall output the recommended systems estimated total price. | Passed |
R5 -> The system shall ask the user the importance of performance of each of the common computer applications. | Passed |
R6 -> The system shall ask the user what they can afford to spend on the new computer system. | Passed |
Expert's Price Estimate | $1700 |
Expert's Build Sheet | AthlonXP 2100+, 400W Case, 1.5GB DDR-RAM, 21" Monitor, Sound Blaster Audigy, 120GB HD, DVD-RW, Generic Speakers, Gigabyte DDR Motherboard, GeForce4 Ti4400 |
Computer Advice Inputs | Use graphics development applications Need a monitor Large monitor Budget $1700 |
Computer Advice Build Sheet | Level Price Description ------------------------------------------------------- 6 $234 Athlon XP 2100+ 0 $44 Enlight Medium ATX Tower w/ 300W PS 6 $255 GeForce4 Ti4400 3 $115 80GB WD 7200 RPM Hard Drive 0 $25 52x CD-ROM Drive 0 $58 16x CD-RW Drive 6 $423 3 x 512MB PC2100 DDR 266MHz (1.5GB Total) 4 $148 Asus A7N266 nForce (DDR, Sound, Video) 0 $11 Microsoft Intelli Mouse PS2 0 $9 Mitsumi 104-key PS2 Keyboard 0 $9 Mitsumi 1.44MB 3.5in Floppy Drive 4 $202 HP Printer/Fax/Scanner/Copier Y40 (8ppm, 600x600) 4 $142 Generic 17" Monitor ------------------------------------------------------- Total: $1675 |
Computer Advice Rating | Agree. |
Expert Comments | Monitor choice is a little off, real world situation would call for a removal of the HP all-in-one device in favor of the 21" monitor, but in this case, a good match. |
Case 7: Software development computer (Medium price point)
Verification Criteria | Passed |
The system shall be free from compilation errors (and warnings), runtime errors, and shall not throw any unhandled exceptions. | Passed |
Given a sufficient budget, the system shall always be able to create a functional computer system. | Passed |
The application shall prompt the user for all of the possible common computer applications. | Passed |
The system shall prompt the user to enter a number for the computer system budget. | Passed |
The system shall print out a computer system build sheet. | Passed |
R1 -> The system shall output the recommended system as a plain text table with each component's description and price. | Passed |
R2 -> The system shall output the recommended systems estimated total price. | Passed |
R5 -> The system shall ask the user the importance of performance of each of the common computer applications. | Passed |
R6 -> The system shall ask the user what they can afford to spend on the new computer system. | Passed |
Expert's Price Estimate | $600 |
Expert's Build Sheet | AthlonXP 2100+ CPU, 1.0GB DDR-RAM, NIC, CD-RW, 340W Case, Gigabyte DDR Motherboard, 40GB HD |
Computer Advice Inputs | Use software development applications Budget $600 |
Computer Advice Build Sheet | Level Price Description ------------------------------------------------------- 2 $75 Duron 1.3 GHz 0 $44 Enlight Medium ATX Tower w/ 300W PS 0 $63 Biostar M7VKG KT133A (PC133, Audio, Video) 1 $63 256MB PC133 SDRAM 0 $48 GeForce2 MX200 0 $20 Standard 56k Winmodem / 100BTX NIC 2 $95 60GB Maxtor 7200 RPM Hard Drive 0 $25 52x CD-ROM Drive 0 $58 16x CD-RW Drive 0 $79 HP Deskjet 845C (8ppm, 600x600) 0 $11 Microsoft Intelli Mouse PS2 0 $9 Mitsumi 104-key PS2 Keyboard 0 $9 Mitsumi 1.44MB 3.5in Floppy Drive ------------------------------------------------------- Total: $599 |
Computer Advice Rating | Agree. |
Expert Comments | My estimate was off on this system. |
Case 8: Gaming and audiophile computer system (High price point)
Verification Criteria | Passed |
The system shall be free from compilation errors (and warnings), runtime errors, and shall not throw any unhandled exceptions. | Passed |
Given a sufficient budget, the system shall always be able to create a functional computer system. | Passed |
The application shall prompt the user for all of the possible common computer applications. | Passed |
The system shall prompt the user to enter a number for the computer system budget. | Passed |
The system shall print out a computer system build sheet. | Passed |
R1 -> The system shall output the recommended system as a plain text table with each component's description and price. | Passed |
R2 -> The system shall output the recommended systems estimated total price. | Passed |
R5 -> The system shall ask the user the importance of performance of each of the common computer applications. | Passed |
R6 -> The system shall ask the user what they can afford to spend on the new computer system. | Passed |
Expert's Price Estimate | $1900 |
Expert's Build Sheet | AthlonXP 2100+ CPU, 1.5GB DDR-RAM, Sound Blaster Audigy Platinum, 16x DVD-ROM, 21" Monitor, 400W Case, GeForce4 Ti4600, 120GB HD, Gigabyte DDR Motherboard, Premium Speakers |
Computer Advice Inputs | Use gaming applications Audiophile system Budget $1900 |
Computer Advice Build Sheet | Level Price Description ------------------------------------------------------- 4 $107 Athlon XP 1800+ 0 $44 Enlight Medium ATX Tower w/ 300W PS 7 $380 GeForce4 Ti4600 0 $20 Standard 56k Winmodem / 100BTX NIC 4 $179 Createive Labs Audigy Platinum 5 $173 120GB WD 7200 RPM Hard Drive 0 $25 52x CD-ROM Drive 5 $400 Klipsch 5.1-THX Speakers (500W Total) 4 $214 256MB + 512MB PC2100 DDR 266MHz (768MB Total) 4 $148 Asus A7N266 nForce (DDR, Sound, Video) 0 $11 Microsoft Intelli Mouse PS2 0 $9 Mitsumi 104-key PS2 Keyboard 0 $9 Mitsumi 1.44MB 3.5in Floppy Drive 4 $142 Generic 17" Monitor ------------------------------------------------------- Total: $1861 |
Computer Advice Rating | Agree. |
Expert Comments | Good match on video card. |
Case 9: Video-editing system (High price point)
Verification Criteria | Passed |
The system shall be free from compilation errors (and warnings), runtime errors, and shall not throw any unhandled exceptions. | Passed |
Given a sufficient budget, the system shall always be able to create a functional computer system. | Passed |
The application shall prompt the user for all of the possible common computer applications. | Passed |
The system shall prompt the user to enter a number for the computer system budget. | Passed |
The system shall print out a computer system build sheet. | Passed |
R1 -> The system shall output the recommended system as a plain text table with each component's description and price. | Passed |
R2 -> The system shall output the recommended systems estimated total price. | Passed |
R5 -> The system shall ask the user the importance of performance of each of the common computer applications. | Passed |
R6 -> The system shall ask the user what they can afford to spend on the new computer system. | Passed |
Expert's Price Estimate | $2300 |
Expert's Build Sheet | Athlon XP2100+ CPU, 1.5GB DDR-RAM, 120GB HD, Pinnacle Studio Deluxe, 400W Case, ATI FireGL, DVD-RW, 21" Monitor, Gigabyte DDR Motherboard |
Computer Advice Inputs | Use video-editing applications Need a monitor Large monitor Budget $2300 |
Computer Advice Build Sheet | Level Price Description ------------------------------------------------------- 6 $234 Athlon XP 2100+ 3 $133 Antec SX1040 Tower w/ 400W PS 0 $20 Standard 56k Winmodem / 100BTX NIC 4 $179 Createive Labs Audigy Platinum 5 $173 120GB WD 7200 RPM Hard Drive 1 $48 16x DVD-ROM Drive 4 $62 16x CD-RW Drive 5 $400 Klipsch 5.1-THX Speakers (500W Total) 5 $290 All-In-Wonder Radeon 8500DV 64MB AGP 6 $423 3 x 512MB PC2100 DDR 266MHz (1.5GB Total) 4 $148 Asus A7N266 nForce (DDR, Sound, Video) 0 $11 Microsoft Intelli Mouse PS2 0 $9 Mitsumi 104-key PS2 Keyboard 0 $9 Mitsumi 1.44MB 3.5in Floppy Drive 4 $142 Generic 17" Monitor ------------------------------------------------------- Total: $2281 |
Computer Advice Rating | Strongly Agree. |
Expert Comments | Good match. Video capture could be done through the AIW Radeon 8500. |
Case 10: Child's computer (Low price point)
Verification Criteria | Passed |
The system shall be free from compilation errors (and warnings), runtime errors, and shall not throw any unhandled exceptions. | Passed |
Given a sufficient budget, the system shall always be able to create a functional computer system. | Passed |
The application shall prompt the user for all of the possible common computer applications. | Passed |
The system shall prompt the user to enter a number for the computer system budget. | Passed |
The system shall print out a computer system build sheet. | Passed |
R1 -> The system shall output the recommended system as a plain text table with each component's description and price. | Passed |
R2 -> The system shall output the recommended systems estimated total price. | Passed |
R5 -> The system shall ask the user the importance of performance of each of the common computer applications. | Passed |
R6 -> The system shall ask the user what they can afford to spend on the new computer system. | Passed |
Expert's Price Estimate | $350 |
Expert's Build Sheet | Duron 950MHz CPU, 300W Case, 128MB PC133 RAM, 20GB HD |
Computer Advice Inputs | Budget $350 |
Computer Advice Build Sheet | Level Price Description ------------------------------------------------------- 0 $44 Duron 950 MHz 0 $44 Enlight Medium ATX Tower w/ 300W PS 0 $63 Biostar M7VKG KT133A (PC133, Audio, Video) 0 $38 128MB PC133 SDRAM 0 $65 20GB WD 7200 RPM Hard Drive 0 $25 52x CD-ROM Drive 0 $11 Microsoft Intelli Mouse PS2 0 $9 Mitsumi 104-key PS2 Keyboard 0 $9 Mitsumi 1.44MB 3.5in Floppy Drive ------------------------------------------------------- Total: $308 |
Computer Advice Rating | Strongly Agree. |
Expert Comments | Excellent Results. |