Computer Efficiency Testing Resources by and

In 2019, the European Commission indicated that it would welcome support developing an on-mode test method for computers. CLASP contacted the Commission and offered to assist on this topic, by developing a first version of a software tool that would quantify both energy consumption and performance of different computer configurations across several operating systems. The Commission then invited CLASP and its Team (GTD and Intertek UK) to present their concept to key stakeholders to determine whether this idea should be taken forward.

This page contains links to all resources of the Computer Efficiency Testing Project by GTD, Intertek and CLASP.

Presentations

Webinar 2022

Webinar 2020

The presentations given by GTD, Intertek and CLASP on 7 July 2020 explain the purpose of this project, give an overview about the test setup and software and show our results.

Test Procedure

This document sets out the procedure to follow to test the energy efficiency and performance of a computer. The equipment necessary for conducting the testing is specified as well as all the steps related to hardware setup and software execution which need to be followed by the technician.

Metric White Paper

A white paper which explains the energy efficiency metric chosen for the test software can be found on the link below.

Test Suite Software

Updating an existing USB key

An already existing USB key can be quickly updated to the current version by running the update script as outlined in the Test Procedure §6.4.

Current version 2023-03-17

On the link below, you can download the file required to create a USB key containing the Computer Efficiency Test Software. The file is 12GB in size and needs to be extracted before creating the USB key.

Changes since the previous version:

• Spaceship: Better timeout handling
• Spaceship: Make installation more robust
• Blender: Fix result parsing for test duration >1hour
• Blender: Print log message which hardware accelerator is used
• Print start/end timestamps of each worklet try to logfile
• Disable Chrome auto-updates on Windows even more
• Fix support for Hioki 3332 power meter

Older versions

2023-03-08

On the link below, you can download the file required to create a USB key containing the Computer Efficiency Test Software. The file is 12GB in size and needs to be extracted before creating the USB key.

Changes since the previous version:

• Blender: Detect Intel GPUs (Intel instead of Arc)
• Handbrake: Enable QSV video decoding
• Spaceship: Kill test runner after 10 minutes to prevent endless hangs
• Use default browser for result viewer (instead of hardcoded Chrome)
• Only use best Handbrake result in Perf/Energy metric
• MacOS: Remove dependency to homebrew, instead include pre-built binary packages for all dependencies
• MacOS: Fix fatal PHP error in system monitor
• Chrome OS: Preload chroot with all necessary packages which would otherwise be installed via APT
• Chrome OS: Add workaround for files in /tmp not being executable

2023-02-14

On the link below, you can download the file required to create a USB key containing the Computer Efficiency Test Software. The file is 11GB in size and needs to be extracted before creating the USB key.

Changes since the previous version:

• Updates to individual worklets:
  • Blender: Use v3.4.1 and auto-detect GPU accelerators
  • Handbrake: Use full flatpak for quick sync support on Linux
  • Handbrake: Remove quality check
  • Spaceship: Update to improved version with 4K offscreen rendering
  • FreeCAD: More realistic models and more repetitions
  • FreeCAD: Update to v0.20
  • SQLite: More realistic database operations
  • LibreOffice: Update to v7.4.4
  • PyBench: Update to work on Python v3.10
  • Selenium: Update to Chrome v109
  • Stockfish: Include correct training network in download-cache
• Updates to the test infrastructure:
  • Only include the best Handbrake result in meta efficiency calculation
  • Additional geometric means for performance and power
  • Perf/Energy metrics now use Watt hours instead of Watt seconds
  • Minor improvements to log output
  • Make sure all worklets write to their logfiles instead of standard out
  • Include LibreOffice and FreeCAD packages for MacOSX and Linux, instead of relying on Homebrew and Apt
  • Update to Phoronix Test Suite v10.8.4
  • Update to Python v3.10
  • Update order of worklets
  • Update number of worklet retries
• Removals and clean-ups:
  • Remove need for Python v2.7 on Windows
  • Remove need for Homebrew Cask tap on MacOSX
  • Remove historical code from Test GUI application

2022-09-19

On the link below, you can download the file required to create a USB key containing the Computer Efficiency Test Software. The file is 9GB in size and needs to be extracted before creating the USB key.

Changes since the previous version:

• Update Linux rootfs used on ChromeOS to a Ubuntu LTS version
• Improve time synchronization on MacOSX

2022-07-20

On the link below, you can download the file required to create a USB key containing the Computer Efficiency Test Software. The file is 9GB in size and needs to be extracted before creating the USB key.

Changes since the previous version:

• Run Unity Spaceship Demo always at 1080p resolution
• Reorder worklets to alternate compute-intense and office-type worklets
• Repeat worklets less often to decrease time needed for test runs
• Run ZSTD de/compression multiple times to improve result consistency
• Split LibreOffice worklet into 3 individual worklets
• Update Stockfish worklet to use Stockfish v15
• Remove x264, Xonotic, Viking Village worklets
• Fix power meter connection check on MacOSX

2022-04-12

On the link below, you can download the file required to create a USB key containing the Computer Efficiency Test Software. The file is 11GB in size and needs to be extracted before creating the USB key.

Changes since the previous version:

• Update CSV export result ordering (Meta Values first, Handbrake last)
• Update SQLite worklet to use F_BARRIERFSYNC on MacOSX

2022-04-04

On the link below, you can download the file required to create a USB key containing the Computer Efficiency Test Software. The file is 11GB in size and needs to be extracted before creating the USB key.

Changes since the previous version:

• Add Viking Village worklet
• Update ChromeOS chroot filesystem
• Update Handbrake to run for each available hardware accelerator
• Improve time needed for zstd worklet installation
• Improve robustness of software installation on MacOSX
• Fix Handbrake worklet not working on ChromeOS
• Fix Wildlife Extreme not running on high performance GPU on Windows if multiple GPUs available

2022-03-09

On the link below, you can download the file required to create a USB key containing the Computer Efficiency Test Software. The file is 10GB in size and needs to be extracted before creating the USB key.

Changes since the previous version:

• Add 3Dmark Wildlife worklet
• Add Unity Spaceship worklet
• Add Handbrake worklet, including PSNR quality check of result video
• Replace LibreOffice worklet (now uses Impress, Calc and Writer)
• Update LibreOffice version to 7.3
• Update LibreOffice PDF worklet to increase size of test files
• Update Stockfish worklet to Stockfish 14.1
• Update Stockfish worklet to run AVX version if CPU support available
• Update Blender worklet to use Blender 3.0.1
• Update Blender worklet to run natively on Apple M1
• Update Selenium worklet to Google Chrome 99
• Update MP3 encoding worklet to increase test duration
• Update Cygwin for better Windows 11 compatibility
• Update Chroot image for ChromeOS
• Fix SQLite worklet to produce proper results on MacOSX
• Remove all GIMP worklets
• Remove unpack-linux-7zip worklet
• Remove SQLite 8 thread worklet

2021-11-29

On the link below, you can download the file required to create a USB key containing the Computer Efficiency Test Software. The file is 7GB in size and needs to be extracted before creating the USB key.

Changes since the previous version:

• Use ZSH on MacOSX
• Add new mode which only measures performance (requires no power meter)
• Add necessary things to test ChromeOS in developer mode
• Improve support for Ubuntu 21.04
• Improve stability of encode-mp3 worklet
• Improve stability of all gimp worklets
• Run zstd compression and decompression in separate worklets
• Use popcnt version of Stockfish 14 to support entry-level CPUs
• Minor stability improvements to libreoffice and selenium browser worklets

2021-07-09

On the link below, you can download the file required to create a USB key containing the Computer Efficiency Test Software. The file is 16GB in size.

Changes since the previous version:

• Remove 25 worklets
• Add new worklets stockfish, encode-mp3 and gimp from openbenchmarking.org
• Update worklets zstd, tjbench, selenium
• Update gimp-gtd worklet

2021-03-01

Changes since the previous version:

• Deploy GIMP and Chromium via AppImages on Linux to increase reliabilty of corresponding worklets
• Update Google Chrome/Chromium to version 88
• Add support for Apple MacOSX 11
• Add support for Apple M1 (some worklets and the GUI currently require Rosetta2 to run)
• Update to Phoronix Test Suite 10.2
• Support homebrew installation on Apple M1
• Update iozone, openssl, x264 worklets to support Apple M1
• Pin homebrew to a specific version to reliably install the same software versions
• Allow to add a pause between worklets and worklets tries, configurable via GUI
• Disable two worklets which are not available on Apple MacOSX (can be reenabled from the GUI if wanted)
• Show software version date on GUI
• Add Perf/Energy metric to CSV summary export
• Add support for Yokogawa WT3000 variant
• Show a more prominent warning when worklets fail to run due to missing executables

2021-01-19

Changes since the previous version:

• CSV export fix for worklets which run only once
• More robust offline time synchronization on Windows
• Improve uninstallation functionality
• Make sure Chrome is installed in correct version
• Fix potential crash in UUT GUI

2020-10-28

Changes since the previous version:

• More power meters supported (see updated Test Procedure Document above)
• Fix bug in offline time synchronization on Windows
• Optimize CSV summary export

2020-10-14

Changes since the previous version:

• CSV exports
• Offline Time synchronization between UUT and Controller

2020-09-29

Changes since the previous version:

• Yokogawa WT310(E), WT333 and WT500 support
• Option to select which worklets to run
• Option to run worklets in debug mode
• De-installation functionality

2020-07-06

Feedback and Comments

Any feedback regarding the test procedure and test software is highly welcome. Please post your feedback via the following link:

If you do not yet have a user account for the issue tracker, please contact GTD GmbH. Contact details are listed below.

Contact