Verification and Validation

Michael Wales - NeedMoreHP@yahoo.com
EEL 5874 - Expert Systems and Knowledge Engineering



Version History:
DocumentDateDescription
Verification and Validation V1.0April 15, 2002Initial Document
Verification and Validation V2.0April 17, 2002Addded the test results
Verification and Validation V3.0April 18, 2002Shaun Murphy (the expert) evaluated the test results


Table of Contents


Verification Criteria:

The verification for the required ComputerAdvice system will focus on the application following the correct procedures, without any errors. The verification phase is not concerned about the correctness of the output. The following table lists the verification criteria for the system.

Required System - Command prompt system

  1. The system shall be free from compilation errors (and warnings), runtime errors, and shall not throw any unhandled exceptions.
  2. Given a sufficient budget, the system shall always be able to create a functional computer system. The components that must exist for the system to be functional are as follows: CPU, Case, Motherboard, RAM, HD, Optical, Mouse, KB, Floppy
  3. The application shall prompt the user for all of the possible common computer applications listed below: Gaming, Video-editing, Surf the Internet, Software development, Audiophile, Home office, Multimedia server, Graphics development
  4. The system shall prompt the user to enter a number for the computer system budget.
  5. The system shall print out a computer system build sheet, and then exit without any errors or thrown exceptions.
  6. The system shall meet all the requirements having the keyword "shall" in the requirements document.

For the required command prompt system, all of the above verification requirements need to be completed in full. In each and every test case for the application, all of the above requirements should be satisfied. If any requirement is not met during testing, the system will be examined and corrected, and then tested in full again.

Optional System - Web based system

  1. Once the web-server is started, the application can constantly run and will not require any more administration to continue running.
  2. The system should gracefully handle new client connections, and the server side application should be invisible to the client user. The system should start with an empty set of knowledge for each new connection; no knowledge should remain in memory from previous connections.
  3. The system should be able to handle dropped connections, where the user terminated the client application without completely running through the application.

Since this part of the application is optional, not all of these verification criteria need to be met. If any of the optional system verification criteria fails during testing, an effort will be made to correct the problem. If the time required to fix the system will be large enough to affect the development and testing of the required system, the optional system will remain as-is.


Validation Criteria:

The validation phase concentrates on examining the output of the system for correctness. The system will be investigated to make sure that the output is correct, but also that the reasoning is correct also. Validation also makes sure that then end product is what the end user expected.

The validation is the same for both the required system (command prompt based) and the optional system (web based). The systems only differ in how the user interacts with the system, the knowledge engine is identical for each system. The validation will likely occur with the command prompt system simply because it is faster than the web based system.

The validation methodology used will be a hybrid of informal validation and validation by testing. Informal validation is where the developer of the system would meet with the expert and informally test the system. Validation by testing is where test cases are used to make sure that the system agrees with the expert's results. The will use a hybrid of these two processes because the problem my system tries to solve never has a real definite answer.

The validation process will be as follows:

  1. The expert and I will create 6 test scenarios that represent a broad range of common system applications.
  2. The expert will create an approximate build sheet for the system, and a good price for the user to budget on the entire system.
  3. The system will be given the test scenarios and the budgeted amount for the system.
  4. The expert will compare the build sheet that he created to the build sheet the system created. They are not expected to match. The expert will evaluate the ComputerAdvice results, and give a rating such as: strongly agree, agree, acceptable, disagree, unacceptable.
  5. The facts created by each test case will be documented, and the expert will verify that the reasoning used by the ComputerAdvice system is acceptable.

The ComputerAdvice system shall get ratings of strongly agree, agree, and acceptable. If the expert finds the ComputerAdvice recommendation unacceptable, the knowledge will be tuned to get the desired results, and then retested. If the expert and ComputerAdvice disagree, the reasoning used to get there will be looked at for problems. If there appears to be good reason for the ComputerAdvice recommendation, it will be accepted.


Test Cases:

Case 1: Gaming and Home Office computer (Medium price point)

Verification CriteriaPassed
The system shall be free from compilation errors (and warnings), runtime errors, and shall not throw any unhandled exceptions.Passed
Given a sufficient budget, the system shall always be able to create a functional computer system.Passed
The application shall prompt the user for all of the possible common computer applications.Passed
The system shall prompt the user to enter a number for the computer system budget.Passed
The system shall print out a computer system build sheet.Passed
R1 -> The system shall output the recommended system as a plain text table with each component's description and price.Passed
R2 -> The system shall output the recommended systems estimated total price.Passed
R5 -> The system shall ask the user the importance of performance of each of the common computer applications.Passed
R6 -> The system shall ask the user what they can afford to spend on the new computer system.Passed

Expert's Price Estimate$800
Expert's Build SheetDuron 1.3GHz CPU, 256 MB DDR-RAM, 40GB HD, Gigabyte DDR motherboard, CDROM, 300W Case, GeForce3 Ti200
Computer Advice InputsUse gaming applications
Use home office applications
$800 budget
Computer Advice Build SheetLevel Price Description
-------------------------------------------------------
1 $50 Duron 1.1 GHz
0 $44 Enlight Medium ATX Tower w/ 300W PS
2 $105 GeForce4 MX440
0 $20 Standard 56k Winmodem / 100BTX NIC
1 $64 Creative Labs Audigy 5.1
0 $65 20GB WD 7200 RPM Hard Drive
0 $25 52x CD-ROM Drive
0 $58 16x CD-RW Drive
0 $20 Labtec / Altec Lansing 2pc Speakers
1 $80 HP Deskjet 845C (8ppm, 600x600)
0 $79 HP ScanJet Flatbed 220C (600dpi, 42bit color)
1 $73 256MB PC2100 DDR 266MHz
1 $88 GigaByte 7VKML KM266 (DDR, Sound, Video, NIC)
0 $11 Microsoft Intelli Mouse PS2
0 $9 Mitsumi 104-key PS2 Keyboard
0 $9 Mitsumi 1.44MB 3.5in Floppy Drive
-------------------------------------------------------
Total: $800
Computer Advice RatingStrongly Agree
Expert CommentsGood match for application.

Case 2: Video editing computer (High price point)

Verification CriteriaPassed
The system shall be free from compilation errors (and warnings), runtime errors, and shall not throw any unhandled exceptions.Passed
Given a sufficient budget, the system shall always be able to create a functional computer system.Passed
The application shall prompt the user for all of the possible common computer applications.Passed
The system shall prompt the user to enter a number for the computer system budget.Passed
The system shall print out a computer system build sheet.Passed
R1 -> The system shall output the recommended system as a plain text table with each component's description and price.Passed
R2 -> The system shall output the recommended systems estimated total price.Passed
R5 -> The system shall ask the user the importance of performance of each of the common computer applications.Passed
R6 -> The system shall ask the user what they can afford to spend on the new computer system.Passed

Expert's Price Estimate$1100
Expert's Build SheetAthlon XP 2100+ CPU, 1GB DDR-RAM, 120GB HD, DVD-ROM, DVD-RW, 340W Case, Gigabyte DDR Motherboard, AIW Radeon7500
Computer Advice InputsUse video-editing applications
$1100 budget
Computer Advice Build SheetLevel Price Description
-------------------------------------------------------
4 $107 Athlon XP 1800+
2 $61 Enlight Medium ATX Tower w/ 340W PS
0 $20 Standard 56k Winmodem / 100BTX NIC
3 $66 Creative Labs Audigy 5.1
5 $173 120GB WD 7200 RPM Hard Drive
1 $48 16x DVD-ROM Drive
4 $62 16x CD-RW Drive
2 $60 Creative Labs 4.1 Speakers (6W x 4, 17W Sub)
2 $111 All-In-Wonder 128 Pro 16MB PCI
4 $214 256MB + 512MB PC2100 DDR 266MHz (768MB Total)
4 $148 Asus A7N266 nForce (DDR, Sound, Video)
0 $11 Microsoft Intelli Mouse PS2
0 $9 Mitsumi 104-key PS2 Keyboard
0 $9 Mitsumi 1.44MB 3.5in Floppy Drive
-------------------------------------------------------
Total: $1099
Computer Advice RatingAgree
Expert CommentsWould've liked to see the DVD-RW in this system.

Case 3: Bare bones home office system (Low price point)

Verification CriteriaPassed
The system shall be free from compilation errors (and warnings), runtime errors, and shall not throw any unhandled exceptions.Passed
Given a sufficient budget, the system shall always be able to create a functional computer system.Passed
The application shall prompt the user for all of the possible common computer applications.Passed
The system shall prompt the user to enter a number for the computer system budget.Passed
The system shall print out a computer system build sheet.Passed
R1 -> The system shall output the recommended system as a plain text table with each component's description and price.Passed
R2 -> The system shall output the recommended systems estimated total price.Passed
R5 -> The system shall ask the user the importance of performance of each of the common computer applications.Passed
R6 -> The system shall ask the user what they can afford to spend on the new computer system.Passed

Expert's Price Estimate$1100
Expert's Build SheetAthlon XP 2100+ CPU, 1GB DDR-RAM, 120GB HD, DVD-ROM, DVD-RW, 340W Case, Gigabyte DDR Motherboard, AIW Radeon7500
Computer Advice InputsUse video-editing applications
$1100 budget
Computer Advice Build SheetLevel Price Description
-------------------------------------------------------
4 $107 Athlon XP 1800+
2 $61 Enlight Medium ATX Tower w/ 340W PS
0 $20 Standard 56k Winmodem / 100BTX NIC
3 $66 Creative Labs Audigy 5.1
5 $173 120GB WD 7200 RPM Hard Drive
1 $48 16x DVD-ROM Drive
4 $62 16x CD-RW Drive
2 $60 Creative Labs 4.1 Speakers (6W x 4, 17W Sub)
2 $111 All-In-Wonder 128 Pro 16MB PCI
4 $214 256MB + 512MB PC2100 DDR 266MHz (768MB Total)
4 $148 Asus A7N266 nForce (DDR, Sound, Video)
0 $11 Microsoft Intelli Mouse PS2
0 $9 Mitsumi 104-key PS2 Keyboard
0 $9 Mitsumi 1.44MB 3.5in Floppy Drive
-------------------------------------------------------
Total: $1099
Computer Advice RatingAgree
Expert CommentsThis system might be more affordable with a lower levels on CPU and Hard drive selection.

Case 4: Web-server computer (Medium price point)

Verification CriteriaPassed
The system shall be free from compilation errors (and warnings), runtime errors, and shall not throw any unhandled exceptions.Passed
Given a sufficient budget, the system shall always be able to create a functional computer system.Passed
The application shall prompt the user for all of the possible common computer applications.Passed
The system shall prompt the user to enter a number for the computer system budget.Passed
The system shall print out a computer system build sheet.Passed
R1 -> The system shall output the recommended system as a plain text table with each component's description and price.Passed
R2 -> The system shall output the recommended systems estimated total price.Passed
R5 -> The system shall ask the user the importance of performance of each of the common computer applications.Passed
R6 -> The system shall ask the user what they can afford to spend on the new computer system.Passed

Expert's Price Estimate$500
Expert's Build SheetDuron 950MHz CPU, 300W Case, 512MB PC133 RAM, 120MB HD, CDROM
Computer Advice InputsUse webserver applications
$500 budget
Computer Advice Build SheetLevel Price Description
-------------------------------------------------------
1 $50 Duron 1.1 GHz
2 $61 Enlight Medium ATX Tower w/ 340W PS
0 $48 GeForce2 MX200
0 $20 Standard 56k Winmodem / 100BTX NIC
2 $95 60GB Maxtor 7200 RPM Hard Drive
0 $25 52x CD-ROM Drive
1 $73 256MB PC2100 DDR 266MHz
1 $88 GigaByte 7VKML KM266 (DDR, Sound, Video, NIC)
0 $11 Microsoft Intelli Mouse PS2
0 $9 Mitsumi 104-key PS2 Keyboard
0 $9 Mitsumi 1.44MB 3.5in Floppy Drive
-------------------------------------------------------
Total: $489
Computer Advice RatingAgree.
Expert CommentsA higher amount of PC133 ram (not DDR) is a prefered solution to this application.

Case 5: Home Theatre computer (Medium price point)

Verification CriteriaPassed
The system shall be free from compilation errors (and warnings), runtime errors, and shall not throw any unhandled exceptions.Passed
Given a sufficient budget, the system shall always be able to create a functional computer system.Passed
The application shall prompt the user for all of the possible common computer applications.Passed
The system shall prompt the user to enter a number for the computer system budget.Passed
The system shall print out a computer system build sheet.Passed
R1 -> The system shall output the recommended system as a plain text table with each component's description and price.Passed
R2 -> The system shall output the recommended systems estimated total price.Passed
R5 -> The system shall ask the user the importance of performance of each of the common computer applications.Passed
R6 -> The system shall ask the user what they can afford to spend on the new computer system.Passed

Expert's Price Estimate$1100
Expert's Build SheetAthlonXP 1800+ CPU, 300W Case, 512MB DDR-RAM, GeForce3 Ti200, Sound Blaster Audigy Platinum, 60GB HD, DVD-ROM, Premium Surround Sound Speakers
Computer Advice InputsUse multimedia server applications
Audiophile system
Budget $1100
Computer Advice Build SheetLevel Price Description
-------------------------------------------------------
1 $50 Duron 1.1 GHz
2 $61 Enlight Medium ATX Tower w/ 340W PS
3 $156 GeForce3 Ti200
0 $20 Standard 56k Winmodem / 100BTX NIC
3 $66 Creative Labs Audigy 5.1
5 $173 120GB WD 7200 RPM Hard Drive
1 $48 16x DVD-ROM Drive
4 $62 16x CD-RW Drive
3 $100 Createive Labs 5.1 Speakers (6W x 5, 18W Sub)
4 $214 256MB + 512MB PC2100 DDR 266MHz (768MB Total)
2 $121 ABit NV7M nForce (DDR, Sound, Video, NIC)
0 $11 Microsoft Intelli Mouse PS2
0 $9 Mitsumi 104-key PS2 Keyboard
0 $9 Mitsumi 1.44MB 3.5in Floppy Drive
-------------------------------------------------------
Total: $1100
Computer Advice RatingStrongly Agree
Expert CommentsGood match.

Case 6: Graphics development computer (High price point)

Verification CriteriaPassed
The system shall be free from compilation errors (and warnings), runtime errors, and shall not throw any unhandled exceptions.Passed
Given a sufficient budget, the system shall always be able to create a functional computer system.Passed
The application shall prompt the user for all of the possible common computer applications.Passed
The system shall prompt the user to enter a number for the computer system budget.Passed
The system shall print out a computer system build sheet.Passed
R1 -> The system shall output the recommended system as a plain text table with each component's description and price.Passed
R2 -> The system shall output the recommended systems estimated total price.Passed
R5 -> The system shall ask the user the importance of performance of each of the common computer applications.Passed
R6 -> The system shall ask the user what they can afford to spend on the new computer system.Passed

Expert's Price Estimate$1700
Expert's Build SheetAthlonXP 2100+, 400W Case, 1.5GB DDR-RAM, 21" Monitor, Sound Blaster Audigy, 120GB HD, DVD-RW, Generic Speakers, Gigabyte DDR Motherboard, GeForce4 Ti4400
Computer Advice InputsUse graphics development applications
Need a monitor
Large monitor
Budget $1700
Computer Advice Build SheetLevel Price Description
-------------------------------------------------------
6 $234 Athlon XP 2100+
0 $44 Enlight Medium ATX Tower w/ 300W PS
6 $255 GeForce4 Ti4400
3 $115 80GB WD 7200 RPM Hard Drive
0 $25 52x CD-ROM Drive
0 $58 16x CD-RW Drive
6 $423 3 x 512MB PC2100 DDR 266MHz (1.5GB Total)
4 $148 Asus A7N266 nForce (DDR, Sound, Video)
0 $11 Microsoft Intelli Mouse PS2
0 $9 Mitsumi 104-key PS2 Keyboard
0 $9 Mitsumi 1.44MB 3.5in Floppy Drive
4 $202 HP Printer/Fax/Scanner/Copier Y40 (8ppm, 600x600)
4 $142 Generic 17" Monitor
-------------------------------------------------------
Total: $1675
Computer Advice RatingAgree.
Expert CommentsMonitor choice is a little off, real world situation would call for a removal of the HP all-in-one device in favor of the 21" monitor, but in this case, a good match.

Case 7: Software development computer (Medium price point)

Verification CriteriaPassed
The system shall be free from compilation errors (and warnings), runtime errors, and shall not throw any unhandled exceptions.Passed
Given a sufficient budget, the system shall always be able to create a functional computer system.Passed
The application shall prompt the user for all of the possible common computer applications.Passed
The system shall prompt the user to enter a number for the computer system budget.Passed
The system shall print out a computer system build sheet.Passed
R1 -> The system shall output the recommended system as a plain text table with each component's description and price.Passed
R2 -> The system shall output the recommended systems estimated total price.Passed
R5 -> The system shall ask the user the importance of performance of each of the common computer applications.Passed
R6 -> The system shall ask the user what they can afford to spend on the new computer system.Passed

Expert's Price Estimate$600
Expert's Build SheetAthlonXP 2100+ CPU, 1.0GB DDR-RAM, NIC, CD-RW, 340W Case, Gigabyte DDR Motherboard, 40GB HD
Computer Advice InputsUse software development applications
Budget $600
Computer Advice Build SheetLevel Price Description
-------------------------------------------------------
2 $75 Duron 1.3 GHz
0 $44 Enlight Medium ATX Tower w/ 300W PS
0 $63 Biostar M7VKG KT133A (PC133, Audio, Video)
1 $63 256MB PC133 SDRAM
0 $48 GeForce2 MX200
0 $20 Standard 56k Winmodem / 100BTX NIC
2 $95 60GB Maxtor 7200 RPM Hard Drive
0 $25 52x CD-ROM Drive
0 $58 16x CD-RW Drive
0 $79 HP Deskjet 845C (8ppm, 600x600)
0 $11 Microsoft Intelli Mouse PS2
0 $9 Mitsumi 104-key PS2 Keyboard
0 $9 Mitsumi 1.44MB 3.5in Floppy Drive
-------------------------------------------------------
Total: $599
Computer Advice RatingAgree.
Expert CommentsMy estimate was off on this system.

Case 8: Gaming and audiophile computer system (High price point)

Verification CriteriaPassed
The system shall be free from compilation errors (and warnings), runtime errors, and shall not throw any unhandled exceptions.Passed
Given a sufficient budget, the system shall always be able to create a functional computer system.Passed
The application shall prompt the user for all of the possible common computer applications.Passed
The system shall prompt the user to enter a number for the computer system budget.Passed
The system shall print out a computer system build sheet.Passed
R1 -> The system shall output the recommended system as a plain text table with each component's description and price.Passed
R2 -> The system shall output the recommended systems estimated total price.Passed
R5 -> The system shall ask the user the importance of performance of each of the common computer applications.Passed
R6 -> The system shall ask the user what they can afford to spend on the new computer system.Passed

Expert's Price Estimate$1900
Expert's Build SheetAthlonXP 2100+ CPU, 1.5GB DDR-RAM, Sound Blaster Audigy Platinum, 16x DVD-ROM, 21" Monitor, 400W Case, GeForce4 Ti4600, 120GB HD, Gigabyte DDR Motherboard, Premium Speakers
Computer Advice InputsUse gaming applications
Audiophile system
Budget $1900
Computer Advice Build SheetLevel Price Description
-------------------------------------------------------
4 $107 Athlon XP 1800+
0 $44 Enlight Medium ATX Tower w/ 300W PS
7 $380 GeForce4 Ti4600
0 $20 Standard 56k Winmodem / 100BTX NIC
4 $179 Createive Labs Audigy Platinum
5 $173 120GB WD 7200 RPM Hard Drive
0 $25 52x CD-ROM Drive
5 $400 Klipsch 5.1-THX Speakers (500W Total)
4 $214 256MB + 512MB PC2100 DDR 266MHz (768MB Total)
4 $148 Asus A7N266 nForce (DDR, Sound, Video)
0 $11 Microsoft Intelli Mouse PS2
0 $9 Mitsumi 104-key PS2 Keyboard
0 $9 Mitsumi 1.44MB 3.5in Floppy Drive
4 $142 Generic 17" Monitor
-------------------------------------------------------
Total: $1861
Computer Advice RatingAgree.
Expert CommentsGood match on video card.

Case 9: Video-editing system (High price point)

Verification CriteriaPassed
The system shall be free from compilation errors (and warnings), runtime errors, and shall not throw any unhandled exceptions.Passed
Given a sufficient budget, the system shall always be able to create a functional computer system.Passed
The application shall prompt the user for all of the possible common computer applications.Passed
The system shall prompt the user to enter a number for the computer system budget.Passed
The system shall print out a computer system build sheet.Passed
R1 -> The system shall output the recommended system as a plain text table with each component's description and price.Passed
R2 -> The system shall output the recommended systems estimated total price.Passed
R5 -> The system shall ask the user the importance of performance of each of the common computer applications.Passed
R6 -> The system shall ask the user what they can afford to spend on the new computer system.Passed

Expert's Price Estimate$2300
Expert's Build SheetAthlon XP2100+ CPU, 1.5GB DDR-RAM, 120GB HD, Pinnacle Studio Deluxe, 400W Case, ATI FireGL, DVD-RW, 21" Monitor, Gigabyte DDR Motherboard
Computer Advice InputsUse video-editing applications
Need a monitor
Large monitor
Budget $2300
Computer Advice Build SheetLevel Price Description
-------------------------------------------------------
6 $234 Athlon XP 2100+
3 $133 Antec SX1040 Tower w/ 400W PS
0 $20 Standard 56k Winmodem / 100BTX NIC
4 $179 Createive Labs Audigy Platinum
5 $173 120GB WD 7200 RPM Hard Drive
1 $48 16x DVD-ROM Drive
4 $62 16x CD-RW Drive
5 $400 Klipsch 5.1-THX Speakers (500W Total)
5 $290 All-In-Wonder Radeon 8500DV 64MB AGP
6 $423 3 x 512MB PC2100 DDR 266MHz (1.5GB Total)
4 $148 Asus A7N266 nForce (DDR, Sound, Video)
0 $11 Microsoft Intelli Mouse PS2
0 $9 Mitsumi 104-key PS2 Keyboard
0 $9 Mitsumi 1.44MB 3.5in Floppy Drive
4 $142 Generic 17" Monitor
-------------------------------------------------------
Total: $2281
Computer Advice RatingStrongly Agree.
Expert CommentsGood match. Video capture could be done through the AIW Radeon 8500.

Case 10: Child's computer (Low price point)

Verification CriteriaPassed
The system shall be free from compilation errors (and warnings), runtime errors, and shall not throw any unhandled exceptions.Passed
Given a sufficient budget, the system shall always be able to create a functional computer system.Passed
The application shall prompt the user for all of the possible common computer applications.Passed
The system shall prompt the user to enter a number for the computer system budget.Passed
The system shall print out a computer system build sheet.Passed
R1 -> The system shall output the recommended system as a plain text table with each component's description and price.Passed
R2 -> The system shall output the recommended systems estimated total price.Passed
R5 -> The system shall ask the user the importance of performance of each of the common computer applications.Passed
R6 -> The system shall ask the user what they can afford to spend on the new computer system.Passed

Expert's Price Estimate$350
Expert's Build SheetDuron 950MHz CPU, 300W Case, 128MB PC133 RAM, 20GB HD
Computer Advice InputsBudget $350
Computer Advice Build SheetLevel Price Description
-------------------------------------------------------
0 $44 Duron 950 MHz
0 $44 Enlight Medium ATX Tower w/ 300W PS
0 $63 Biostar M7VKG KT133A (PC133, Audio, Video)
0 $38 128MB PC133 SDRAM
0 $65 20GB WD 7200 RPM Hard Drive
0 $25 52x CD-ROM Drive
0 $11 Microsoft Intelli Mouse PS2
0 $9 Mitsumi 104-key PS2 Keyboard
0 $9 Mitsumi 1.44MB 3.5in Floppy Drive
-------------------------------------------------------
Total: $308
Computer Advice RatingStrongly Agree.
Expert CommentsExcellent Results.

Reviewed by Shaun Murphy, 4/17/02
ShaunMurphy.com [shaunmurphy.com]
for Michael Wales