Analysis of large quantities of data requires large computing resources coupled with sophisticated software. The “V’s” of Big Data are important points to consider when implementing an analysis system:

• Volume – how much data is there?
• Velocity – how fast is new data coming in?
• Variety – how many different formats are there to deal with?
• Veracity – how is correctness of the data ensured?

Data

Do you have a large existing pool of data requiring analysis? If so, you will need access to large computing resources to get the job done in a reasonable time.

How much new data do you have coming in every day? You will need sufficient resources to process the new data as quickly as it comes in. If you can project the rate of increase of the amount of data coming in, you can plan for expansion to keep the analysis going.

Transformation of different data formats into a single, consistent data format requires analysts to program the transformation and computers to do the work on an ongoing basis. Testing the correctness of the data requires analysts to build checks into the programs to ensure that the algorithms are generating meaningful results.

Our Services Include:

• Large Scale Computing and Visualization System Consulting
• Visualization
• Numerical and visualization programming
• Data transformation

Large Scale Computing and Visualization System Consulting

For any hardware/software environment, we provide services beginning with initial definition of system requirements, through deployment, and concluding with scheduled maintenance of all the components. We also provide advice on operational processes as needed. See our Large Scale Computing page for more details.

Computing hardware:

• Cluster Numerical computing
  • GPGPU’s
  • FPGA’s
  • Hadoop – text analysis
  • visualization cluster
  • distributed file systems (Lustre, ZFS)
• Shared memory processing
  • in memory data analysis
• render farm
• visualization desktop, possibly with multiple graphics cards

Videre Analytics will assist with design of a system to meet your organization’s unique analysis needs.

Visualization

Interactive visualization with large data sets requires both powerful computing and display systems. The ability to work interactively is important because long processing and display refresh times interrupt effective work flow. Thorough data exploration requires visualization work to be done interactively for optimal results to be achieved. Videre Analytics will assist in implementing a system to handle your data visualization needs.

In addition to configuring hardware visualization systems, Videre Analytics has expertise in computer graphics techniques and visualization software. This is discussed on our Visual Analytics page.

Visual data analysis can decrease the time it takes to understand large quantities of data.

Numerical and Visualization Programming

Data analysis requires programming to implement a processing pipeline. We have expertise in a wide range of programming languages and environments, for example:

• Programming languages
  • C/C++
  • Fortran
  • Python
  • Matlab
  • R
  • TCL/Tk
• Programming environments
  • Visual Studio (Windows)
  • Intel Fortran
  • gfortran (Linux and Windows)
• Legacy Fortran programs
  • Maintenance of older versions of Fortran, or
  • translation of legacy Fortran programs to a modern Fortran or another language

Working with scripting and programming languages and modular visualization packages, we can construct analysis and visualization pipelines to meet your requirements.

Data Transformation

Videre Analytics works with data in a variety of formats, in any number of dimensions:

• Numerical or text
• Binary or ASCII
• Database or file-based data sets.

We can work with scientific data whether the data is in a standard format, such as HDF or custom user-defined formats.

3D data sets may be stored in various image formats; Videre Analytics has expertise with tomography data from a number of fields.

We work effectively with large data sets and translate data from one format to another. We can extract and work with data from a variety of sources such as XML, relational databases, APIs, and web services.

When data sets are comprised of various data types, they can be transformed/combined into a single data set for effective processing.