EMMAWiki/NewsAndUpdates: Difference between revisions

Latest revision as of 14:20, 26 October 2011

News and Updates

-- Updates 2011

Support for Agilent High-density arrays
- New analysis functions
- Updated M/A plot
- Improved data browsing and export
- Many more bug fixes

-- Updates 2010

A new home for the EMMA server: http://emma.cebitec.uni-bielefeld.de
- Improved performance

-- Updates 2009

Bugfixes, improvements of usability

-- MichaelDondrup DateTime(2008-09-09T13:14:39Z) EMMA production server updated to version 2.8

Support for NimbleGen Arrays
- Added a converter script for NDF to ADF array layout files
- Added analysis functions and importers from ArrayLIMS
- New Nimblegen Pipelines:
  - Nixpresso

      - RMA
      - Li & Wong
      - limma
      - clustering

Adapted the layout to fit to smaller windows
Made the installation procedure smoother (more variables, create a project)
Create a web-server config fragment automatically using virtual hosts

-- MichaelDondrup DateTime(2008-03-26T13:06:36Z)

EMMA production server updated to version 2.6

New Affymetrix chip pipelines:
- MAS5
- RMA
- GCRMA
- Li & Wong
- Statistical tests
Some improvements in the web-interface
- Configurable messageses on the login and Welcome pages
- Arraylims import got smoother usability and displays more information
Most important changes underneath the surface:
- Configuration management improved using config files in share/config (using the AppConfig module)
- Logging improved using Log4Perl
- Installation procedure improved
- Many bug-fixes

-- MichaelDondrup DateTime(2007-09-07T09:33:51Z)

The EMMA production webserver has been updated to EMMA 2.4

Changing times: EMMA 2.4 has just been released and there are a few points that you should be aware of to ease the transition from EMMA 2.2

1. Please use the most recent version of Firefox. Older Mozilla browsers and Internet explorer browsers will not work properly. Please also enable cookies, pop-up-windows, and Java script, without this nothing will work. Also, make sure that your browser setting support the tool tips that come up when you mouse over certain areas of the EMMA website. These contain some helpful features that you might need. In case you have problems e.g. with opening pop-up windows during import, please check your browser settings and your Java setup, the problem in almost all cases resides there.

2. There are no changes with respect to logins and passwords and also the main interface looks pretty similar.

3. All previous experiments are of course still visible. Note that the colour code has changed: Finished experiments are now in RED. These can only be edited after the project chief has set them back to "in progress". Finishing implies (1) access to all project members on a read-only basis, (2) protection against accidental deleting.

4. The most important changes are connected to experiment setup. Here, a new import wizard takes you through the process of setting up a new experiment. There are a couple of differences that were introduced to achieve MAGE-export compatibility. I summarize the main steps to help you getting started:

page 1:

you now need to enter Firstname, Lastname and email address. The experiment type has now to be defined via the Ontology browser and this definition is mandatory. Under Biological Property you will find most suitable categories, just tick one (or more) and press "Set selected". Please also have a look into all five categories to see if other categories might suit better in your case. You may add multiple categories as you see fit.

page 2:

also for ADD FACTOR, you are now forced to use the Ontology browser. "EnvironmentalFactorCategory" is e.g. suitable for temperature, etc., "BiologicalFactorCategory"/"BioMaterialCharacteristicCategory" will work e.g. for strain or line or organism part, pathogen treatment, etc. Please browse as long as needed to find a suitable factor. Then press "set selected". You can now define values by pressing "ADD VALUE" (multiple times for multiple values). Here, you can enter free text, but make sure to also click on "No Measurement" to select an appropriate unit, e.g. degr_centigrade for a temperature factor, and do not to write the unit e.g. °C again in the text field. The more ontology-based your description is, the better any export will work. Please take some time to do this as good as you can, it pays off in the end!

page 3:

this page forces you to describe the experiment. No description, no further import!

page 4:

you can now "SAVE WITHOUT IMPORT" (if you want to import your arrays later using the menu on the left or if you already have arrays in the database from other experiments that you want to connect to a new experiment) or "SAVE AND START IMPORT". This opens the import dialogue.

page 5-9:

import as in EMMA 2.2. Remember to select the experimental and control changes in the correct orientation (data from the experimental channels have to be selected on the left and give positive M-values). Press FINISH.

5. Link arrays: Once you have imported your arrays, you have to press link arrays (under "Analysis results" on the bottom of the experiment main page. Now you get a new design for the connection of factors and values to arrays. Just drag and drop values and arrays to build your experiment (from my personal experince it helps a lot to have this ready on a sheet of paper....). If you find this confusing, just select Link arrays(oldschool) on the left. This is not that posh, but works equally well and it is actually the same interface that you already know from Emma 2.2.

Please be aware that you don't have to import the same array multiple times in multiple experiments. In case you have already imported arrays in an old experiment and want to link them also to a new one, just click in "Link arrays" on the experiment main page, go to "Unlinked Arrays" / "Add and remove existing arrays used in the current project" / "BROWSE ARRAYS" / tick the arrays and press "close window". Then the arrays show up under "unlinked arrays" (btw: the same process reversed allows you to remove unwanted arrays from an experiment).

Remember that there should be no unlinked arrays in your experiment (otherwise you cannot start an analysis any more!) and that you can assign the same array to different factors in one experiment. To avoid unlinked array assign at least one value for each factor to each array. You can check this in the FactorBrowser. There must not be a RED unlinked column for valid experiments.

Please try to use a multi-factor setup in the future to properly describe the experimental factors (instead of defining mixed factor/values, e.g. do not define a factor "strain-and-time-test" with two values "strain1_vs_2_at_10dpi" and "strain3_vs_2_at_20dpi" but factor one "strain" with values 1vs2 and 3vs2 AND factor "time" with values 10dpi and 20dpi; etc. The better you adhere to this, the better any future export will be.

6. The old "edit" interface is gone and is replaced by an interface from the import wizard.

7. The "Add StickyNote" feature is a playground for the project administrator (Chief). I can use this e.g. to ask users for permissions to finish experiments or to request modifications, etc.

I hope you find the modifications and this small tutorial useful!

Updated documentation is available under

http://www.cebitec.uni-bielefeld.de/groups/brf/software/emma_info/docu.html

In EMMA, please use the "Help/Submit a bug" feature to report technical bugs, problems and suggestions directly to the emma development team or use the contact email address mentioned on the login screen. Please describe any bugs as detailed as you can and with all information needed to reproduce it.

(Thanx to Helge Küster for this text)

Further changes

Experiment page: Plots, graphics, cluster analyses and other files which are generated by pipelines are displayed as attachements which can be opened directly by clicking the icons.
Welcome page: Enhanced icons and text. If you are a project administrator you may also go to the configure project page from the start page.
Web-services: EMMA is also a web-service client. Whenever you look at sequence data, EMMA checks if that seuqence is linked to a Web-service (aka. CoryneRegNet)

database. The dat from the external web-service database is retrieved and displayed. For CoryneRegNet for example, this provides information on the regulatory network and the operon in which the gene is contained. Please note that the availability of this feature depends on your project.

Data browser: The web-service feature is also available for data in the data browser. Click the get web-service information button to retrieve the info for all sequences on that page. Due to timing constraints it is impossible to do this automatically for all sequences. Avoid retrieval for more than a few hundred sequences!
MA-plot: This function now offers additional plot types such as density color plots. Therefore, the configuration dialog offers more options, such as color scheme for the density colors.
Vioplot: This is a new function which allows to plot the data distribution for a given experiment for all KEGG pathways. Please note that the availability of this feature depends on your project. Your project needs to be linked to GenDB/SAMS, the genome annotation needs to contain EC information, and the Vioplot function needs to be added to your EMMA project.

Old changes (EMMA 2.2)

-- MichaelDondrup DateTime(2006-02-18T17:36:01Z)

The Reporter Datamining Tool allows to search all experiments within a Project for transformed data. The tool allows to search for Reporter names and identifiers using wildcards and also it allows for Google style fulltext search in Descriptions. The resulting datamatrix may be exported as tab-separated file. Have a look at Reporter Datamining in the Advanced Actions menue. Please note that the maximum number of Reporters to be displayed is limited to 400.
Cluster Viewer: Now, you can resort the data in the Show reporters window. Click on the columnheader to sort by that column, click more often to toggle descending, ascending and no sorting. You can also sort by Reporter Name and Description.
A new history function allows to go back several steps at once. It is locate at the top of the main window. It's working with most of the pages now,

but in rare cases the link in the history might go wrong, displaying an empty page. Errors can also occur if try to go back to a deleted object. No data will be lost as delete operations are not stored in the history.

Chiefs can configure the links to external databases like EMBL, TREMBL, TIGR. Use the Configure DB links dialogue to edit these settings. This is most useful if in the Sequence Details there is no hyperlink to the external resource. For eg. TIGR medicago gene index the link is: http://www.tigr.org/tigr-scripts/tgi/tc_report.pl?species=medicago&tc=
! The Advanced Actions menue now opens and closes only on click not on mouse over. The old behaviour was too annoying.
The login page checks for activated Cookies and JavaScript. If one of them is not activated or the user tries to use InternetExplorer, a warning message is displayed
Cluster browser: Cluster browser now shows Factor/Values, you can activate a Factor/Value label for the tree view also
Cluster browser has new auto-expand option, zooming the whole tree to have readable row-labels
Setting the status of an Experiment to Locked or Finished now protects the Experiment against accidential deletion, even for roles Chief and higher
Normal users may only edit an Experiment in Status Locked if they own it. Only Chiefs and higher may edit an Experiment in Status Finished. This is necessary to make deletion of finished Experiments possible: Set status back to In progress
Data Viewer: The Filter dialog now gives you the option to build boolean queries using AND and OR. Queries can be grouped by parenthesis '()'.

You still need to define filter conditions. They are named E1 ... En. A sample boolean query looks like this:  { (E1 AND E2) OR E3 }
See feature request: https://bugs.cebitec.uni-bielefeld.de/show_bug.cgi?id=511

The layout of the filter dialog is changed, take a look at the AND and OR buttons at the end of the line. They will add an entry to the expression field.

New Filters operators within interval and not within interval. Define a range by two numbers.
Array View: Now the red-green overlay images can be created on the fly. You may specify gamma, contrast and brightness. This only works for newly imported images. (Cutspotpix program is no longer needed) according to feature request https://bugs.cebitec.uni-bielefeld.de/show_bug.cgi?id=517
If you click on a spot in the zoom window, now, Reporter details will be displayed.
Pipeline: EMMA can now do Normalization on Controls, the necesary functions have been added to the system. (https://bugs.cebitec.uni-bielefeld.de/show_bug.cgi?id=528) Like always: If you need Normalization on Controls in your project, you have to request the function to be added.
Hierarchical clustering now uses the amap package instead of hclust. The cluster pipeline should now be more memory efficient and also faster, for large datasets
Manual data-set selection under Advanced-options shows more information
Experiment: The number of clusterings performed is listed on the overview page
Array Design: Link to documentation is now fixed https://bugs.cebitec.uni-bielefeld.de/show_bug.cgi?id=523
Object Permissions: Now shows dependent object by default, when adding a permissions set https://bugs.cebitec.uni-bielefeld.de/show_bug.cgi?id=498
Search: You can now also search descriptions
KEGG browser: You can view the available KEGG maps under advanced options. Expression data can be projected on these pathways. Therefor, an analysis step is required, which computes a GeneExpressionMatrix from the normalized data. In the results page you will find a link to the KEGG projection. The necessary pipelines will be added to these projects.

-- MichaelDondrup DateTime(2005-12-27T13:36:13Z) (from Testing DateTime(2005-09-22T11:17:01Z))

Documentation: links to HTML help pages can now be found in the info-text (on the left) on almost each page
Cluster browser (Java applet) is improved (This applet deserves its own page, in fact)
- You can cut the tree into clusters
- Plot line-graphs, bar and star charts of expression profiles
- Zoom into any region
- Change colors
- Export the graphics as image files and data files
- ...
Cluster page: speed up in loading the tree and matrix data: data is cached, reloading a cluster will be much faster, now
Clusters can now be deleted, and have their permissions changed
Clusters are now under the control of the permission system
Array layout: The ArrayLayout import is now more verbose, when a problem with missing or unknown columns is discovered.
Experiment Wizzard: More information (name and identifier) under link arrays
Pipeline: The significance test can now do a two.sample test, the old one, was lacking the parameter. To profit from that, a new pipline has to be imported.
Experiment page: More editable descriptions (Quality control, Normalization and Replication)

-- MichaelDondrup DateTime(2005-09-07T16:06:39Z)

Renamed this page, updated links to it to EMMAWiki/NewsAndUpdates. Now other documentations may have NewsAndUpdates, too.

-- MichaelDondrup DateTime(2005-09-07T15:55:43Z) (from Testing: DateTime(2005-08-30T17:59:41Z))

slight improvement of the layout and function of the Dataset Viewer. Filter Data button moved to the top of the page. Drop sorting can now be re-done, and the drop button is only displayed, when data was filtered or sorted. Note: the Drop sorting button will also drop a filtering step.

-- MichaelDondrup DateTime(2005-08-19T14:58:48Z)

LIMMA[[FootNote(Smyth, G. K., Yang, Y.-H., Speed, T. P. (2003). Statistical issues in microarray data analysis. In: Functional Genomics: Methods and Protocols, M. J. Brownstein and A. B. Khodursky (eds.), Methods in Molecular Biology Volume 224, Humana Press, Totowa, NJ, pages 111-136.)]] statistical test functions should now be available in every project. LIMMA is a (probably) superior alternative to a simple t-test for identifying differentially expressed genes. Chiefs may create a new pipeline with this function. For more information, refer to the LIMMA user's guide by the authors of the R-package.

-- MichaelDondrup DateTime(2005-08-19T14:41:48Z) (from Testing: DateTime(2005-08-17T20:41:06Z))

added possibility to change owner of an object under View permissions B-) only for Chiefs B-)
added a 'More help' link under the info box on a few pages
added icon to menu->bug report
edit and delete buttons are now displayed only when the user has approriate right, to perform the action

-- MichaelDondrup DateTime(2005-08-19T14:41:48Z) (from Testing: DateTime(2005-08-15T08:11:31Z))

Added link to news and updates to the login pages
Adjusted the maximum length of Name and Identifier fields in Edit Experiment page to 400 and 200.
Added a utility to add and edit groups under 'Settings->Manage Groups' in the top menu.

-- MichaelDondrup DateTime(2005-08-19T14:41:48Z) (from Testing: DateTime(2005-08-15T08:11:31Z))

Improved 'Experiment->View permissions' can now also change permissions of dependent object (aka: Arrays, Datasets)
Pipeline should now be more efficient and require less memory, speed up of normalization. It should now be possible to normalize more than 100 arrays in parallel.
Cluster browser now shows Factor/Value headers in '(right click on tree node)->show reporters', when available. If not, please re-run the clustering to see them. The Factor/Value headers will also be present in the 'export data-matrix' export

Testing

-- MichaelDondrup DateTime(2008-03-26T13:06:36Z)

Webserver updated to EMMA 2.6 (release)

-- MichaelDondrup DateTime(2008-01-30T09:44:30Z)

Webserver updated to EMMA 2.5 (alpha development)
New Affymetrix array pipelines:
- MAS5
- RMA
- GCRMA
- Li & Wong
- Statistical tests
Some improvements in the web-interface
- Configurable messageses on the login and Welcome pages
- Arraylims import got smoother usability and displays more information
Most important changes underneath the surface:
- Configuration management improved using config files in share/config (using the AppConfig module)
- Logging improved using Log4Perl
- Installation procedure improved
- Many bug-fixes

-- MichaelDondrup DateTime(2006-02-18T17:36:01Z)

Webserver has been updated to EMMA 2.2(final)
There was a small bug in the Data Viewer (Filter-function) causing not all Expressions to be evaluated

-- MichaelDondrup DateTime(2006-01-24T10:34:57Z)

Webserver has been updated to EMMA 2.2 Release Candidate 2

Bugfixes

Stable

-- MichaelDondrup DateTime(2008-04-02T14:00:00Z)

Bug in the Arraylims import wizard caused an exception when slide information in ArrayLIMS is deleted (bug 798).
Bug in the ArraylimsImporter caused failed import jobs.
When deleting an experiment, a wrong link was displayed (bug 790)

-- MichaelDondrup DateTime(2006-10-04T08:04:14Z)

GenePix import: Despite the Imagene importer, the Genepix importer did not respect the channel order of the assigned data files. For Imagene, there are always two data files, which can be easily exchanged. But for Genepix, and most other software there is only a single data file. For Genepix imports, the channel assignments of the data files are now checked for their orientation and dye swaps will be automatically marked as 'swapped' data sets. This was preferred over simply echanging the data between F532 and F635 columns of the imput file, because this procedure would have caused inconsistency with the original input files.
Analysis Pipeline: Resulting from the previous fix, it was reported, that the analysis functions raised errors when run on such a swapped dataset. This was caused by a bug in the mapping table for Genepix data types.

The above errors where reported and fixed thanks to reports from Delphine Capela.

-- MichaelDondrup DateTime(2006-09-06T09:55:00Z)

General security: There was a bug the delete mechanism, that would lead to accidentially deleting wrong objects (Experiments) under rare circumstances. This bug was fixed, the error condition will now raise an Internal Error.
Database links: The mechanism which protects EMMA against cross-site scripting attacks (by blocking HTML entry into any text field) was overly restrictive. It did not allow for entry of correct URIs for external database links. This is now fixed. Note that entry of any HTML tags is still impossible for all fields.
Sequence Viewer (GenDB links): GenDB BRIDGE links did only function for GenDB regions of type CDS. This was partially due to a problem in GenDB and also in EMMA's Sequence viewer module. Sequence annotation from gendb will now be displayed correctly for any region type if it exists. Now all functions View Contig, Jump to Region, and View Annotation should also work.
Data Import: The GenePix importer could fail with GenePix files from Genepix versions <= 2 (Though, we never said these files were supported ). Before, it did work in principle. Now, this should in principle always work...

This NewsAndUpdates transmission ws brought to you by Radio Eriwan :D

-- MichaelDondrup DateTime(2006-03-21T09:29:05Z)

Sequence Viewer (GenDB links): BRIDGE-links to GenDB via View Contig, Jump to Region, and View Annotation did not work correctly. The direkt links to GenDB have been fixed (see Bug 612). Still, due to some deficiencies in GenDB, only the link 'Jump to Region' will work correctly (see GenDB Bug 620 and GenDB Bug 621)

-- MichaelDondrup DateTime(2006-03-20T10:47:27Z)

Cluster Viewer: There was a minor bug, that caused the image export of the tree to be disfunct, see Bug 608. Re-starting your browser might be necessary.

-- MichaelDondrup DateTime(2006-02-23T17:39:02Z)

There was a little bug, causing the Genepix import to be disfunct. (Bug 596)

-- MichaelDondrup DateTime(2006-02-18T17:36:01Z)

There was a bug in the filter settings of the FilterHClust module, causing overly strict filtering on hierarchical clustering pielines.

-- MichaelDondrup DateTime(2006-01-26T18:57:31Z)

SignificanceTest: There was a bug in the two-sample SignificanceTest function causing false values for the T-Statistic and p-values. The one-sample test was not affected by this bug. see https://bugs.cebitec.uni-bielefeld.de/show_bug.cgi?id=573

-- MichaelDondrup DateTime(2006-01-24T10:34:57Z)

Cluster Viewer: small bug fixed in the search window. That bug would cause the text entry field to grow proportionally when resizing the window.
Normalization on Controls (new in 2.2rc1) had a bug that prevented it from being effective. It took all the data and computed median or mean normalization, not only the control set.
Analysis Pipeline: The reconfiguration dialogue displayed by clicking on Change configuration in Analyse

Arrays had a bug that would cause multi-valued parameters (like the filter flags in the Normalization pipeline) to be ingnored except for the first value. The predefined pipelines were not affected by that. (See https://bugs.cebitec.uni-bielefeld.de/show_bug.cgi?id=550)

Delete selected did not ask for confirmation, now you will have to confirm a deletion (https://bugs.cebitec.uni-bielefeld.de/show_bug.cgi?id=540)
The speed of the Data Viewer has been significantly improved. When the number of Features in a project increased, the function we used could become very slow, even causing a 502 Proxy Error. (https://bugs.cebitec.uni-bielefeld.de/show_bug.cgi?id=561)

-- MichaelDondrup DateTime(2005-12-28T10:45:57Z)

Data Viewer: The enable reporter detail button dropped the sorting of the table (https://bugs.cebitec.uni-bielefeld.de/show_bug.cgi?id=500)
Data Viewer: There was a little bug, preventing the ReporterControlTypes from being displayed
Data Viewer - export -: There was a bug when exporting a Feature-based Dataset. The data for ReporterControlType where missing, shifting the whole table to the left by one column
Link arrays wizzard: There was a bug which allowed the user to add an array twice to the same experiment (https://bugs.cebitec.uni-bielefeld.de/show_bug.cgi?id=484)
Filter Data: entering a comma (1,1) instead of dot (1.1) in a filter option would show erratic behaviour. Now commas are replaced by dots. (https://bugs.cebitec.uni-bielefeld.de/show_bug.cgi?id=501)
ArrayLayoutImport: There was a bug in the tab-separated-file-parser, causing a left shift for columns containg empty columns. (https://bugs.cebitec.uni-bielefeld.de/show_bug.cgi?id=448)
ArrayLayoutImport: There was a bug, preventing ADF files containing special characters like 'é, è, Â, °' to be imported. This was due to an encoding problem. While the uploaded file is encoded using the locale of the web-browser, the XML-file needs UTF-8 encoding. Now, the data is recoded to UTF-8, before creating the XML. (https://bugs.cebitec.uni-bielefeld.de/show_bug.cgi?id=371)

-- MichaelDondrup DateTime(2005-12-30T15:13:27Z)

There was a tiny bug in the Pipeline configuration module, preventing the proper assignment of array parameters for functions.See https://bugs.cebitec.uni-bielefeld.de/show_bug.cgi?id=550

-- MichaelDondrup DateTime(2005-12-27T13:36:13Z) (from Testing DateTime(2005-09-22T11:17:01Z))

Fixed a bug, that would cause more trouble than neccessary for chiefs, when trying to delete objects they did not own
Fixed a bug in the Pipeline, that would cause mixed Factor Value assignments and lead to wrong results, when more than one Factor was defined.
Changed confusing links to 'Create experiment (wizzard)'
Changed confusing help links
Removed a bug that prevented the two sample t-test from working
Typos...

-- MichaelDondrup DateTime(2005-09-07T15:55:43Z) (from Testing: DateTime(2005-08-30T17:54:44Z))

Fixed a bug in a new feature of the Data Viewer, that would cause an exception when trying to sort a dataset.

-- MichaelDondrup DateTime(2005-08-19T14:41:48Z) (from Testing: DateTime(2005-08-17T20:41:06Z))

fixed a bug in ArrayLimsImport now access rights should always be respected
News and Updates now also works in logout page

-- MichaelDondrup DateTime(2005-08-19T14:41:48Z) (from Testing: DateTime(2005-08-15T08:11:31Z))

Fixed a bug, that would cause several Arrays/ datasets not to be added to the current experiment in rare cases, during parallel import from arraylims, or parallel execution of multiple pipelines.
Fixed a minor bug on the cluster list page, now the return to link is functional, while viewing descriptions.

-- MichaelDondrup DateTime(2005-08-19T14:41:48Z) (from Testing: DateTime(2005-08-15T11:49:36Z))

Removed a bug that would break arraylims import in rare cases. When the uploaded data file has been processed under a Windows/DOS operating system,sometimes CR/LF will be added as line-break indicator. The linebreak indicators are now converted to UNIX style during parsing.
Made the ArrayLayoutImport more robust, against empty lines. Now the layout import will ignore empty input lines and also remove quotes (aka: "").

EMMAWiki/NewsAndUpdates: Difference between revisions

Latest revision as of 14:20, 26 October 2011

News and Updates

Further changes

Old changes (EMMA 2.2)

Testing

Bugfixes

Stable

Navigation menu

Search