Marvin’s Change Log¶
[2.5.3] - 2020/16/1¶
Changes - static web images to webp format for better compression and faster load times.
[2.5.2] - 2020/10/1¶
Restructuring to simplify setup, dependencies, and tests, for code health.
Adds - New
setup.py
andsetup.cfg
to simplify installation and dependency requirementsAdds - New dependencies
marvin-brain
,marvin-wtforms-alchemy
,marvin-sqlalchemy-boolean-search
,sdss-tree
, andsdss-access
.Changes - Migrates to
sdsstools
for internal logging and config management.Changes - Removes all external submodules, which have been converted into pypi releases.
Changes - Moves tests outside the python package
[2.5.1] - 2020/09/25¶
Fixes #711 - galaxy page bug when NSA dictionary not available prevents flag modals from appearing
[2.5.0] - 2020/09/04¶
Adds - support for MaNGA data release MPL-10
Removes - web / API support for MPLs 4, 5, 6
Removes - SPX and VOR10 binning schemese in MPLs 7-10, and DR15-16.
[2.4.1] - 2020/08/05¶
Adds - support for using the marvin mirror at magrathea.sdss.org, hosted at Johns Hopkins
Adds - new
use_mirror
custom config option to automatically set marvin to use the mirror
[2.4.0] - 2020/07/19¶
Adds - caching web routes on Plate, Galaxy pages
Adds - new
use_cache
option toCachable Query
class. Implemented in most marvin database calls.Adds - most Marvin Tools now use query caching for database calls
Adds - a new
CacheMixIn
that adds optional caching to classes utilizing db callsAdds - a new
Redis
dogpile.cache
backend; with new regions forMaps
andModelcubes
Adds - new utility functions for DRPall, and DAPall files,
get_xxx_path
,get_xxx_file
,get_xxx_table
Adds - adds visual morphology VAC for DR16+
Fixes - Issue #695 - better web spectra/map error messages;
Changes - tree and sdss_access submodules to latest 3.0+ and 1.0+ versions
Changes - path changes to accommodate new tree and sdss_access
[2.3.6] - 2020/04/14¶
[2.3.5] - 2019/12/09¶
Added¶
Support for MPL-9 MaNGA data
Issue #648 - Option to toggle common lines from the DAP in web spectrum viewer
Issue #112 - Option to toggle between rest-frame and observed wavelength in web spectrum viewer
Issue #679 - Option to toggle on DONOTUSE pixel region masks in web spectrum viewer
master-fifo file to uwsgi base config ini file
adds an alert to authenticate when no static image detected
Changed¶
updated datamodels to include MPL-9 changes
turned off API rate limiting globally
turned back on old spaxel query method
switched random image web page to use improved
get_random_images
functionchanging galaxy page static image grab to use
cube.getImage().url
[2.3.4] - 2019/12/09¶
Changed¶
Added explicit python path in uwsgi prod.ini file
Fixed¶
Bug in session variable controlling DAP bintemps
[2.3.3] - 2019/12/09¶
Added¶
Support for DR16. Updated datamodels and docs.
Added Galaxy Zoo VAC
Added GEMA VAC
Added Firefly VAC
Added Visual Morphology VAC
Merged #678, richer support for VACs
Merged #652, richer support for Windows
Merged #677, updated documentation on issues with fuzzy string matching
Merged #687, fixes map arithmetic with reflexive operators
Changed¶
all yaml.load uses new Loader to accommodate old and new yaml spec;
updated Runtime Issues documentation to include section on numpy.ufunc binary warnings
added to reprs for some db ModelClasses
all
sdss_access.sync.RsyncAccess
has been converted tosdss_access.sync.Access
to allow for Linux/Windows switchingupdates citation to Marvin paper
DAP Maps label on galaxy page now contains a pop up of links for more info
Fixed¶
Issue #658 - modelcube datamodel units missing angstrom
Issue #655 - added yaml loader to remove yaml warnings for 5.1 spec
Fixed bug when server tries to access NSA on cube when none exists and triggers remote authentication issue
Issue #664 - Fixed link to MaNGA’s Getting Started in docs.
Issue #686 - added reflexive operators to EnhancedMap
Issue #689 - added link to CSV file of Firefly stellar mass measurements needed for the Spatially-Resolved Mass-Metallicity Relation Tutorial
Issue #682 - adds info links next to DAP Maps selection on galaxy page
Issue #581 - makes MapSpecView button a session variable
Issuu #208 - makes map selections session variable
Issue #171 - fix y-min to 0 for spectrum web view
[2.3.0] - 2018/12/03¶
Breaking changes¶
Removed
Bin
class. Bin information is now available on a per-quantity basis (#109). See What’s new? and documentation for details.Syntax on the inputs to the
Query
andResults
tools has been changed.DAP spaxel queries have been disabled due to performance issues. We expect to bring them back soon. Metadata queries (those querying the NSA or DAPall tables) are still available.
getSpaxel
now only loads the quantities from the parent object (that means that, for example,Maps.getSpaxel
only loadsMaps
properties by default). Additional quantities can be loaded usingload
.getSpaxel
accepted arguments have been changed tocube
,maps
, andmodelcube
. The formerly accepted arguments (drp
,properties
,model(s)
) now raise a deprecation error.
Added¶
Added cheatsheet to docs.
New Web authentication using Flask-Login
New API authentication using Flask-JWT-Extended
Adds MPL-7 / DR15 datamodel
New config.access attribute indicating public or collab access
New config.login method to handle token-based login
New marvin.yml config file for customization of configuration options
Adds User table into the history schema of mangadb. Tracks user logins.
Map
has a new methodgetSpaxel
to retrieve anSpaxel
using the parentMaps
.New configuration option in
marvin.yml
,default_release
, to set the release to be used when Marvin gets imported (#463).Applying a NumPy ufunc (except
np.log10()
) raisesNotImplementedError
because ivar propagation is not implemented yet.New
Marvin Image
Tool to load optical images using the MMA (#22)New
Bundle
andCutout
utility classesNew
MMAMixIn
for providing multi-modal data accessqual_flag
andpixmask_flag
are now stored in the datamodel (#479, #482).Query
tool now accepts a newtargets
andquality
keyword argument which enables querying on target or quality maskbit labels. (#485)Added a new API route for streaming large query results. This uses a generator to stream large results back to the client to minimize memory usage and bulk responses.
Changed¶
Integrated datamodel plotting params into actual datamodel structures
Moved netrc checks into the Brain
Expanded sasurl into public and collab urls
Changes personal emails to sdss helpdesk email in web
Added rawsql and return_params columns to history.query table in mangadb
Extra keyword arguments passed to
Spectrum.plot
are now forwarded toAxes.plot
.Tools (e.g.,
Cube
,Maps
) can now be accessed from themarvin
namespace (e.g.,marvin.tools.Cube
ormarvin.tools.cube.Cube
).Map plotting
ax_setup()
function is now hidden.Moved
yanny.py
toextern/
and added a readme file for the external packages (#468).plot
now only masks part of the spectrum that have theDONOTUSE
maskbit set (#455).pixmask
is now available for all quantities (exceptAnalysisProprty
). The propertymasked
now uses the bitDONOTUSE
to determine what values must be masked out (#462).Raises error when applying
inst_sigma_correction
onstellar_sigma
MPL-6 maps. Applies correction to stellar_sigma and emline_sigma for web maps with added ‘Corrected’ title (#478)Removes targeting bits from
Spaxel
andBin
(#465).The name of the channel is now shown in the
Property
description (#424).Replaced inconsistent parameter
model
ingetSpaxel
. Usemodels
instead.MarvinError now accepts an optional
ignore_git
keyword to locally turn off the git addition to the messageUsing the
return_all
keyword inQuery
orgetAll
inResults
now calls the streaming API route instead.When
Cube
orModelCube
are instantiated from file, gunzip the file to a temporary location to speed up subsequent access (#525).Convert MMA warnings to debug messages (:issues:`580`).
Fixed¶
Issue #421 - query returning spaxel props returns wrong total count
Bugfix - Python 3 xrange syntax bug in buildImageDict
Bin._create_spaxels
instantiating spaxels with the wrong(i,j)
values for the bin. The(i, j)
values from thebinid
map were being swapped twice before sending them toSpaxelBase
(#457).A bug in the calculation of the deredden inverse variance in a
DataCube
.Issue with setting drpall path on initial import/set of release before tree has been planted
Issue #456 - spectrum web display shows incorrect RA, Dec
Issue #422 - ensuring config auto checks access to netrc
Issue #423 - adding marvin.yml documentation
Issue #431 - adding login documentation
Issue #151 - adding web spectrum tooltips
Issue #548 - failed to retrieve ModelCube extension in remote mode
Fixed typo by in method name
Spectrum.derredden -> Spectrum.deredden
.Fixed #305 - adding ivar propogation for np.log10(Map)
A bug when explicitly returning default parameters in a query (#484)
Fixed #510 - fixes incorrect conversion to sky coordinates in map plotting.
Fixed #563 - fail retrieving Query datamodels in Python 3.6+.
Fixes bug with sasurl not properly being set to api.sdss.org on initial import
Incorrect setting of the default bintype to download from web (#531).
Incorrect mismatch warning between MPL-7 and DR15 (#495).
Incorrect handling of maskbits when the mask does not contain any of the bits in the list (#507).
Fixed #534 - flipped axes in NSA scatterplot when plotting absmag colors
Fixed #559 - bug in check_marvin when marvindb is None
Fixed #579 - bug in MMA with marvindb preventing files from opening
Fixed #575 - cannot access maps due to bug in login and authentication in Interaction class
Fixed #539 - print downloadList target directory
Fixed #566 - made error message for web query with non-unique parameters name more specific
Refactored¶
Moved
marvin.core.core
tomarvin.tools.core
and split the mixins intomarvin.tools.mixins
.Reimplemented
getAperture
as a mixin using photutils apertures (#3, #315).Reimplemented
RSS
as a list ofRSSFiber
objects (#27, #504).Moved pieces of MarvinToolsClass into
marvin.tools.mixins
.Reimplemented
Query
to remove local query dependencies from remote mode usage.
[2.2.5] - 2018/04/26¶
Changed¶
Galaxy Web page spaxel loading to be robust when no modelspaxels are present in the database.
[2.2.3] - 2018/03/20¶
Added¶
Added tests for
emline_gflux_ha_6564
and fixed values in galaxy_test_data.
Fixed¶
Issue #182
Issue #202
Issue #319
Issue #322
Issue #334
Issue #339
Issue #341
Issue #342
Issue #348
Issue #352
Issue #354
Issue #355
Issue #362
Issue #366
Issue #367
Issue #368
Issue #369
Issue #372
Issue #375
Issue #378
Issue #379
Issue #383
Issue #385
Issue #386
Issue #374: Cube units do not persist under axis reordering.
Fixed some problems with test_spaxel tests.
Issue #382: Is fuzzywuzzy too fuzzy?
Fixed an issue with Astropy 3 in
get_nsa_data()
.Fixed some issues with query results tests
Issue #391
Issue #387
Issue #384
Issue #380
Issue #376
Issue #373
Issue #371
Issue #370
Issue #363
Issue #361
Issue #360
Issue #359
Issue #358
Issue #357
Issue #353
Issue #351
Issue #349
Issue #346
Issue #345
Issue #344
Issue #343
Issue #340
Issue #337
Issue #336
Issue #335
Issue #333
Issue #331
Issue #330
Issue #329
Issue #328
Issue #327
Issue #326
Issue #325
Issue #324
Issue #320
Issue #307
Issue #395
Issue #390
Removed¶
The banner that showed up in Safari has been removed since most versions should now work properly.
[2.2.2] - 2018/02/25¶
Fixed¶
MPL-6 issue with all H-alpha extensions mapped to NII instead. Indexing issue in MPL-6 datamodel.
MPL-6 issue with elliptical coordinate extensions; missing R/Reff channel in MPL-6 datamodel.
Issue #324
Issue #325
Issue #326
Issue #327
Issue #330
Issue #333
Issue #335
Issue #336
Issue #343
Issue #351
Issue #353
Issue #357
Issue #358
Issue #360
Issue #363
Issue #373
[2.2.0] - 2018/01/12¶
Added¶
Added
Maskbit
class for easy conversion between mask values, bits, and labels.Better BPT documentation, in particular in the
Modifying the plot
section.A hack function
marvin.utils.plot.utils.bind_to_figure()
that replicate the contents of a matplotlib axes in another figure.New scatter and histogram plotting utility functions
Integrated scatter and histogram plotting into query Results
New methods for easier query Results handling
New Pythonic DRP, DAP, and Query DataModels
Access to DAPall data
Changed¶
Issue #190:
Maps.get_bpt()
andmarvin.utils.dap.bpt.bpt_kewley06()
now also return a list of axes. Each axes contains a method pointing to themarvin.utils.plot.utils.bind_to_figure()
function, for easily transfer the axes to a new figure.All Cubes, Maps, and Modelcubes use Astropy Quantities
Refactored to the Bin class
Bin and Spaxel are now subclassed from SpaxelBase
Fixed¶
Issue #24
Issue #99
Issue #110
Issue #111
Issue #131
Issue #133
Issue #173
Issue #178
Issue #180
Issue #190
Issue #191
Issue #233
Issue #235
Issue #246
Issue #248
Issue #261
Issue #263
Issue #269
Issue #279
Issue #281
Issue #286
Issue #287
Issue #290
Issue #291
Issue #294
Issue #295
Issue #296
Issue #297
Issue #299
Issue #301
Issue #302
Issue #303
Issue #304
Issue #308
Issue #311
Issue #312
[2.1.4] - 2017/08/02¶
Added¶
Added new query_params object, for easier navigation of available query parameters. Added new tests.
Added a new guided query builder using Jquery Query Builder to the Search page
Added a View Galaxies link on the web results to view postage stamps of the galaxies in the results
Added Route Rate Limiting. Adopts a limit of 200/min for all api routes and 60/minute for query api calls and web searches
Changed¶
Changed call signature for :meth:
marvin.utils.plot.map.no_coverage_mask
(removedvalue
arg because unused, addedNone
as default valueivar
(None
), and re-ordered args and kwargs).Changed call signature for :meth:
marvin.utils.plot.map.bad_data_mask
(removedvalue
arg because unused).Changed the Marvin web search page to use the new query_params and parameter grouping. Removed the autocomplete input box.
Updated the documentation on query and query_params.
Modified Guided Search operator options to remove options that could not be parsed by SQLA boolean_search
Refactored the web settings, route registration, extensions to enable extensibility
Issue #282: Improvements to “Go to CAS” link. Changed to Go To SkyServer and updated link to public up-to-date link
Fixed¶
Issue #102: problem with urllib package when attempting to retrieve the Marvin URLMap
Issue #93: safari browser does not play well with marvin
Issue #155: Contrails in Web Map
Issue #174: sdss_access may not be completely python 3 compatible
Issue #196: Bin not loading from local sas
Issue #207: Get Maps in MapSpecView of Galaxy page sometimes fails to return selected maps
Issue #210: pip upgrade may not install new things as fresh install
Issue #209: marvin version from pip install is incorrect
Issue #268: Cube flux from file error
Issue #85: Python does not start in Python 3
Issue #273: ha.value bug
Issue #277: Ticks for log normalized colorbar
Issue #275: logger crashes on warning when other loggers try to log
Issue #258: 422 Invalid Parameters
Issue #271: Problem in dowloading image.
Issue #97: sqlalchemy-boolean-search not found when installed from pip source
Issue #227: Marvin installation in python 3.6 (update setuptools to 36)
Issue #262: problem with marvin update
Issue #270: BPT array sizing not compatible
Issue #88: Deployment at Utah requires automatisation
Issue #234: Add (and use) functions to the datamodel to determine plotting parameters
Issue #278: marvin_test_if decorator breaks in python 2.7
Issue #274: cube slicing to get a spaxel fails with maps error
Issue #39: implement more complete testing framework
Issue #242: Result object representation error with 0 query results
Issue #159: Marvin issues multiple warnings in PY3
Issue #149: Improve integrated flux maps display in web
[2.1.3] - 2017/05/18¶
Added¶
Issue #204: added elpetro_absmag colours to mangaSampleDB models.
Issue #253: Plotting tutorial.
Issue #223: Easy multi-panel map plotting (with correctly placed colorbars).
Issue #232 and Issue #251: Uses matplotlib style sheets context managers for plotting (map, spectrum, and BPT) and restores previous defaults before methods finish.
Issue #189: Map plotting accepts user-defined value, ivar, and/or mask (including BPT masks).
Issue #252: Quantile clipping for properties other than velocity, sigma, or flux in web.
Added
utils.plot.map
doc page.Added
tools.map
doc page.
Changed¶
Issue #243: inverted
__getitem__
behaviour for Cube/Maps/ModelCube and fixed tests.Modified Flask Profiler File to always point to $MARVIN_DIR/flask_profiler.sql
Issue #241: Moved map plotting methods from tools/map to utils/plot/map
Issue #229 and Issue #231: Switch to new gray/hatching scheme (in tools and web):
gray: spaxels with NOCOV.
hatched: spaxels with bad data (UNRELIABLE and DONOTUSE) or S/N below some minimum value.
colored: good data.
Issue #238: Move plot defaults to datamodel (i.e., bitmasks, colormaps, percentile clips, symmetric, minimum SNR).
Issue #206: SNR minimum to None (effectively 0) for velocity maps so that they aren’t hatched near the zero velocity contour.
Simplified default colormap name to “linearlab.”
Decreased map plot title font size in web so that it does not run onto second line and overlap plot.
Fixed¶
Interactive prompt for username in sdss_access now works for Python 3.
Fixed #195: The data file for the default colormap for
Map.plot()
(“linear_Lab”) is now included in pip version of Marvin and does not throw invalidFileNotFoundError
if the data file is missing.Fixed #143: prevents access mode to go in to remote if filename is present.
Fixed #213: shortcuts are now only applied on full words, to avoid blind replacements.
Fixed #206: no longer masks spaxels close to zero velocity contour in web and tools map plots
Fixed #229: corrects web bitmask parsing for map plots
Fixed #231: hatch regions within IFU but without data in map plots
Fixed #255: Lean tutorial code cells did not work with the ipython directive, so they now use the python directive.
Highcharts draggable legend cdn.
[2.1.2] - 2017/03/17¶
Added¶
API and Web argument validation using webargs and marshmallow. If parameters invalid, returns 422 status.
Changed¶
Per Issue #186: Switched to using the elpetro version of stellar mass, absolute magnitude i-band, and i-band mass-to-light ratio for NSA web display, from sersic values. (elpetro_logmass, elpetro_absmag_i, elpetro_mtol_i)
Issue #188: deprecated snr in favour of snr_min for get_bpt. snr can still be used.
Issue #187: Renamed NSA Display tab in web to Galaxy Properties. Added a link to the NASA-Sloan Atlas catalogue to the table title.
Moved our documentation to readthedocs for version control. Updated all Marvin web documenation links to point to readthedocs.
[2.1.1] - 2017/02/18¶
Added¶
Added query runtime output in search page html. And a warning if query is larger than 20 seconds.
Changed¶
Removed the python 3 raise Exception in the check_marvin bin
Reverted the api/query return output from jsonify back to json.dumps
This is an issue with python 2.7.3 namedtuple vs 2.7.11+
[2.1.0] - 2017/02/16¶
Added¶
Restructured documentation index page.
Improved installation documentation:
Removed old installation text
Added section on marvin SDSS dependencies and SAS_BASE_DIR
Added section for FAQ about installation
Added web browser cache issue into FAQ
Added traceback info in the API calls
Added traceback attribute in Brain config
Added hidden _traceback attribute in Marvin config
Only implemented in two Query API calls at the moment
Added a few tests for traceback
see usage in cube_query in marvin/api/query.py
Added the Ha_to_Hb ratio the DAP ModelClasses for querying
Added new script to perform somce basic system, os, and Marvin checks: bin/check_marvin
Added an alert banner when the user is using Safari. See #94.
Issue #122: added ra/dec to spaxel
Issue #145: Limited the number of query parameters in the web
Added more tests to Results for sorting, paging, and getting subsets
Added kwargs input for Spaxel when using Result.convertToTool
Added automatic Sentry error logging #147 into MarvinError, and Sentry in Flask for production mode
Added custom error handlers for the web, with potential user feedback form
Added Sentry tool for grabbing and displaying Sentry statistics
Added text to MarvinError with a Github Issues link and description of how to submit and issue
Added Results option to save to CSV
Added new parameters in Marvin Config to turn off Sentry error handling and Github Issue message
Added Python example code for getting a spectrum in galaxy page of web.
Added new test for image utilities getRandomImages, getImagesByPlate, getImagesByList
Added new documentation on Image Utilities
Added new image utility function showImage, which displays images from your local SAS
Added the Kewley+06 implementation of the BPT classification as
Maps.get_bpt()
Added quick access to the NSA information for a Cube/Maps either from mangaSampleDB or drpall.
Changed¶
When marvin is running from source (not dist),
marvin.__version__
isdev
.Removed the cleanUpQueries method to assess db stability
Switched dogpile.cache from using a file to python-memcached
Syntax changes and bug fixes to get Marvin Web working when Marvin run on 3.5
Got Queries and Results working in 3.5
Changed all convertToTool options in Results from mangaid to plateifu
Added release explicitly into api query routes
Modified the decision tree in query to throw an error in local mode
Modified convertToTool to accept a mode keyword
Modifed the MarvinError for optional Sentry exception catching, and github issue inclusion
Updated all Marvin tests to turn off Sentry exception catching and the github message
Updated some of the Tools Snippets on the web
Overhauled Map plotting
uses DAP bitmasks (NOVALUE, BADVALUE, MATHERROR, BADFIT, and DONOTUSE)
adds percentile and sigma clipping
adds hatching for regions with data (i.e., a spectrum) but no measurement by the DAP
adds Linear Lab color map
adds option for logarithmic colorbar
adds option to use sky coordinates
adds map property name as title
makes plot square
sets plotting defaults:
cmap is linear_Lab (sequential)
cmap is RdBu_r (diverging) for velocity plots (Note: this is reversed from the sense of the default coolwarm colormap in v2.0—red for positive velocities and blue for negative velocities)
cmap is inferno (sequential) for sigma plots
clips at 5th and 95th percentiles
clips at 10th and 90th percentiles for velocity and sigma plots
velocity plots are symmetric about 0
uses DAP bitmasks NOVALUE, BADVALUE, MATHERROR, BADFIT, and DONOTUSE
also masks spaxels with ivar=0
minimum SNR is 1
Changed Marvin Plate path back to the standard MarvinToolsClass use
Made sdss_access somewhat more Python 3 compatible
Modified the image utilities to return local paths in local/remote modes and url paths when as_url is True
downloadList utility function now downloads images
updated the limit-as parameter in the uwsgi ini file to 4096 mb from 1024 mb for production environment
Fixed¶
Issue #115: drpall does not get updated when a tool sets a custom release.
Issue #107: missing os library under save function of Map class
Issue #117: hybrid colours were incorrect as they were being derived from petroth50_el.
Issue #119: test_get_spaxel_no_db fails
Issue #121: bugfix with misspelled word in downloadList utility function
Issue #105: query results convertToTool not robust when null/default parameters not present
Issue #136: BinTest errors when nose2 run in py3.5 and marvin server in 3.5
Issue #137: PIL should work in py2.7 and py3.5
Issue #172: broken mode=auto in image utilities
Issue #158: version discrepancy in setup.py
[2.0.6] - 2016/11/17¶
Fixed¶
Bug in Queries with dap query check running in remote mode. Param form is empty.
[2.0.5] - 2016/11/17¶
Added¶
Added netrc configuration to installation documentation.
Added netrc check on init.
Fixed¶
Added mask to model spaxel.
Bug in Cube tool when a galaxy loaded from db does not have NSA info; no failure with redshift
Two bugs in index.py on KeyErrors: Sentry issues 181369719,181012809
Bug on plate web page preventing meta-data from rendering
Fixed installation in Python 3.
Fixed long_description in setup.py to work with PyPI.
Fixed a problem that made marvin always use the modules in extern