{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Persistent Archival of Python Objects"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "[![Documentation Status](https://readthedocs.org/projects/persist/badge/?version=latest)](https://persist.readthedocs.io/en/latest/?badge=latest)\n",
    "[![Language grade: Python](https://img.shields.io/lgtm/grade/python/g/forbes-group/persist.svg)](https://lgtm.com/projects/g/forbes-group/persist/context:python)\n",
    "[![Tests](https://github.com/forbes-group/persist/actions/workflows/tests.yml/badge.svg)](https://github.com/forbes-group/persist/actions/workflows/tests.yml)\n",
    "[![Pypi](https://img.shields.io/pypi/v/persist.svg)](https://pypi.python.org/pypi/persist)\n",
    "[![pyversions](https://img.shields.io/pypi/pyversions/persist.svg)](https://pypi.python.org/pypi/persist)[![Code style: black](https://img.shields.io/badge/code%20style-black-000000.svg)](https://github.com/psf/black)\n",
    "\n",
    "Persistent archival of python objects in an importable format.\n",
    "\n",
    "This package provides a method for archiving python objects to disk for long-term persistent storage.  The archives are importable python packages with large data stored in the [npy](https://docs.scipy.org/doc/numpy/neps/npy-format.html) numpy data format, or [HDF5](http://www.hdfgroup.org/HDF5/) files using the [h5py](http://www.h5py.org) package (if it is installed). The original goal was to overcomes several disadvatages of pickles:\n",
    "\n",
    "1. Archives are relatively stable to code changes.  Unlike pickles, changing the underlying code for a class will not change the ability to read an archive if the API does not change.\n",
    "2. In the presence of API changes, the archives can be edited by hand to fix them since they are simply python code.  (Note: for reliability, the generated code is highly structured and not so \"pretty\", but can still be edited or debugged in the case of errors due to API changes.)\n",
    "3. Efficient storage of large arrays.\n",
    "4. Safe for concurrent access by multiple processes.\n",
    "\n",
    "**Documentation:**\n",
    "   http://persist.readthedocs.org\n",
    "\n",
    "**Source:**\n",
    "   https://alum.mit.edu/www/mforbes/hg/forbes-group/persist\n",
    "   \n",
    "**Issues:**\n",
    "   https://alum.mit.edu/www/mforbes/hg/forbes-group/issues"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Installing"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This package can be installed from [PyPI](https://pypi.org/project/persist/):\n",
    "\n",
    "```bash\n",
    "python3 -m pip install persist\n",
    "```\n",
    "\n",
    "or from source:\n",
    "\n",
    "```bash\n",
    "python3 -m pip install hg+https://alum.mit.edu/www/mforbes/hg/forbes-group/persist\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# DataSet Format"
   ]
  },
  {
   "cell_type": "raw",
   "metadata": {
    "raw_mimetype": "text/restructuredtext"
   },
   "source": [
    ".. toctree::\n",
    "   :maxdepth: 1\n",
    "   \n",
    "   notebooks/DataSet Format"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# API"
   ]
  },
  {
   "cell_type": "raw",
   "metadata": {
    "raw_mimetype": "text/restructuredtext"
   },
   "source": [
    ".. toctree::\n",
    "   :maxdepth: 3\n",
    "\n",
    "   api/persist"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Developer Notes"
   ]
  },
  {
   "cell_type": "raw",
   "metadata": {
    "raw_mimetype": "text/restructuredtext"
   },
   "source": [
    ".. toctree::\n",
    "   :maxdepth: 1\n",
    "\n",
    "   notebooks/Pickle\n",
    "   notebooks/Dev Notes"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Release Notes"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As of version 3.1, we release only to PyPI using [`poetry`](https://python-poetry.org/).  Here is the typical development/release cycle.\n",
    "\n",
    "* First make sure you have a development environment with Mercurial, the evolve extension, topics enabled, [Black], [Nox], and [nbconvert].\n",
    "\n",
    "* Set your virtual environment and run a shell to work in:\n",
    "\n",
    "   ```bash\n",
    "   poetry env use python3.8\n",
    "   poetry shell\n",
    "   poetry install -E doc -E test\n",
    "   ```\n",
    "\n",
    "* Start a development branch, i.e.:\n",
    "\n",
    "    ```bash\n",
    "    hg branch 3.2\n",
    "    ```\n",
    "* Change version to `'3.2.dev0'` in `pyproject.toml` and commit this changes:\n",
    "\n",
    "    ```bash\n",
    "    hg com -m \"BRN: Start working on branch 3.2\"\n",
    "    hg push --new-branch -r . \n",
    "    ```\n",
    "  \n",
    "* Complete your changes making sure code is well tested etc.  While working on specific features, you should always use topics:\n",
    "\n",
    "    ```bash\n",
    "    hg topic new-feature\n",
    "    ```\n",
    "    \n",
    "    When you push to Heptapod, the commits in these topics will remain in the draft phase, allowing you to rebase, etc. as needed to clean the history.  We have setup automatic pushes to [GitHub](https://github.com/forbes-group/persist) and you can see the status of the tests with the badge: [![Github Tests](https://github.com/forbes-group/persist/actions/workflows/tests.yml/badge.svg)](https://github.com/forbes-group/persist/actions/workflows/tests.yml).\n",
    "    \n",
    "    To run the tests locally, you should be able to just run:\n",
    "    \n",
    "    ```bash\n",
    "    nox\n",
    "    ```\n",
    "    \n",
    "* Once everything is working and tested, push it to Heptapod and create Merge Requests:\n",
    "\n",
    "  * First merge all open topics to the development branch.\n",
    " \n",
    "* Then change the revision in `pyproject.toml` to `'3.2'`, dropping the `'.dev'`.  Push this to Heptapod and create a merge request to merge this to the default branch.  Review the changes, and complete the Merge.  Unlike previously, **do not close the branch.**  Just leave it.\n",
    "  \n",
    "* Start work on next branch::\n",
    "\n",
    "    ```bash\n",
    "    hg up 3.2\n",
    "    hg branch 3.3\n",
    "    ```\n",
    "\n",
    "## PyPI\n",
    "To release on PyPI:\n",
    "\n",
    "```bash\n",
    "poetry build\n",
    "poetry\n",
    "\n",
    "    hg up 3.2\n",
    "    poetry build\n",
    "    \n",
    "    python setup.py sdist bdist_wheel\n",
    "    twine upload dist/persist-3.2*\n",
    "\n",
    "## Anaconda Cloud\n",
    "To release on Anaconda Cloud (replace the filename as appropriate):\n",
    "\n",
    "    conda build meta.yaml\n",
    "    anaconda upload --all /data/apps/conda/conda-bld/osx-64/persist-3.0-py37_0.tar.bz2"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Indices and Tables"
   ]
  },
  {
   "cell_type": "raw",
   "metadata": {
    "raw_mimetype": "text/restructuredtext"
   },
   "source": [
    "* :ref:`genindex`\n",
    "* :ref:`modindex`\n",
    "* :ref:`search`"
   ]
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "celltoolbar": "Raw Cell Format",
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.8"
  },
  "toc": {
   "base_numbering": 1,
   "nav_menu": {
    "height": "84px",
    "width": "252px"
   },
   "number_sections": true,
   "sideBar": true,
   "skip_h1_title": false,
   "title_cell": "Table of Contents",
   "title_sidebar": "Contents",
   "toc_cell": false,
   "toc_position": {},
   "toc_section_display": "block",
   "toc_window_display": false
  },
  "varInspector": {
   "cols": {
    "lenName": 16,
    "lenType": 16,
    "lenVar": 40
   },
   "kernels_config": {
    "python": {
     "delete_cmd_postfix": "",
     "delete_cmd_prefix": "del ",
     "library": "var_list.py",
     "varRefreshCmd": "print(var_dic_list())"
    },
    "r": {
     "delete_cmd_postfix": ") ",
     "delete_cmd_prefix": "rm(",
     "library": "var_list.r",
     "varRefreshCmd": "cat(var_dic_list()) "
    }
   },
   "types_to_exclude": [
    "module",
    "function",
    "builtin_function_or_method",
    "instance",
    "_Feature"
   ],
   "window_display": false
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}