Drop SHARED_SECRET in favour of EMAIL_SECRET

Originally we used SHARED secret both for email and for the API.  That
was a bad idea, and now that we're only using this value for one case,
I've renamed it to reflect its actual use.

docs/changelog.rst

@@ -1,6 +1,17 @@
 Changelog
 #########
 
+* 0.6.0
+  * Abandon the shared-secret trick we were using for the POST API in favour
+    of BasicAuth or Django session.
+  * Fix the POST API so it actually works.  `#236`_
+  * **Breaking change**: We've dropped the use of ``PAPERLESS_SHARED_SECRET``
+    as it was being used both for the API (now replaced with a normal auth)
+    and form email polling.  Now that we're only using it for email, this
+    variable has been renamed to ``PAPERLESS_EMAIL_SECRET``.  The old value
+    will still work for a while, but you should change your config if you've
+    been using the email polling feature.  Thanks to `Joshua Gilman`_ for all
+    the help with this feature.
 * 0.5.0
   * Support for fuzzy matching in the auto-tagger & auto-correspondent systems
     thanks to `Jake Gysland`_'s patch `#220`_.
@@ -11,7 +22,8 @@ Changelog
     thanks to `CkuT`_ for finding this shortcoming and doing the work to get
     it fixed in `#224`_.
   * All of the following changes are thanks to `David Martin`_:
-    * Bumped the dependency on pyocr to 0.4.7 so new users can make use of Tesseract 4 if they so prefer (`#226`_).
+    * Bumped the dependency on pyocr to 0.4.7 so new users can make use of
+    Tesseract 4 if they so prefer (`#226`_).
     * Fixed a number of issues with the automated mail handler (`#227`_, `#228`_)
     * Amended the documentation for better handling of systemd service files (`#229`_)
     * Amended the Django Admin configuration to have nice headers (`#230`_)
@@ -206,6 +218,7 @@ Changelog
 .. _CkuT: https://github.com/CkuT
 .. _David Martin: https://github.com/ddddavidmartin
 .. _Paperless Desktop: https://github.com/thomasbrueggemann/paperless-desktop
+.. _Joshua Gilman: https://github.com/jmgilman
 
 .. _#20: https://github.com/danielquinn/paperless/issues/20
 .. _#44: https://github.com/danielquinn/paperless/issues/44
@@ -243,4 +256,5 @@ Changelog
 .. _#228: https://github.com/danielquinn/paperless/pull/228
 .. _#229: https://github.com/danielquinn/paperless/pull/229
 .. _#230: https://github.com/danielquinn/paperless/pull/230
+.. _#236: https://github.com/danielquinn/paperless/issues/236
 

docs/consumption.rst

@@ -125,7 +125,7 @@ So, with all that in mind, here's what you do to get it running:
    ``PATHS AND FOLDERS`` and ``SECURITY``.
    If you decided to use a subfolder of an existing account, then make sure you
    set ``PAPERLESS_CONSUME_MAIL_INBOX`` accordingly here.  You also have to set
-   the ``PAPERLESS_SHARED_SECRET`` to something you can remember 'cause you'll
+   the ``PAPERLESS_EMAIL_SECRET`` to something you can remember 'cause you'll
    have to include that in every email you send.
 3. Restart the :ref:`consumer <utilities-consumer>`.  The consumer will check
    the configured email account at startup and from then on every 10 minutes
@@ -147,46 +147,83 @@ So, with all that in mind, here's what you do to get it running:
 HTTP POST
 =========
 
-You can also submit a document via HTTP POST.  It doesn't do tags yet, and the
+You can also submit a document via HTTP POST, so long as you do so after
-URL schema isn't concrete, but it's a start.
+authenticating.  To push your document to Paperless, send an HTTP POST to the
-
+server with the following name/value pairs:
-To push your document to Paperless, send an HTTP POST to the server with the
-following name/value pairs:
 
 * ``correspondent``: The name of the document's correspondent.  Note that there
   are restrictions on what characters you can use here.  Specifically,
-  alphanumeric characters, `-`, `,`, `.`, and `'` are ok, everything else it
+  alphanumeric characters, `-`, `,`, `.`, and `'` are ok, everything else is
   out.  You also can't use the sequence ` - ` (space, dash, space).
 * ``title``: The title of the document.  The rules for characters is the same
   here as the correspondent.
-* ``signature``: For security reasons, we have the correspondent send a
+* ``document``: The file you're uploading
-  signature using a "shared secret" method to make sure that random strangers
-  don't start uploading stuff to your server.  The means of generating this
-  signature is defined below.
 
 Specify ``enctype="multipart/form-data"``, and then POST your file with::
 
     Content-Disposition: form-data; name="document"; filename="whatever.pdf"
 
+An example of this in HTML is a typical form:
 
-.. _consumption-http-signature:
+.. code:: html
 
-Generating the Signature
+    <form method="post" enctype="multipart/form-data">
-------------------------
+        <input type="text" name="correspondent" value="My Correspondent" />
+        <input type="text" name="title" value="My Title" />
+        <input type="file" name="document" />
+        <input type="submit" name="go" value="Do the thing" />
+    </form>
 
-Generating a signature based a shared secret is pretty simple: define a secret,
+But a potentially more useful way to do this would be in Python.  Here we use
-and store it on the server and the client.  Then use that secret, along with
+the requests library to handle basic authentication and to send the POST data
-the text you want to verify to generate a string that you can use for
+to the URL.
-verification.
-
-In the case of Paperless, you configure the server with the secret by setting
-``UPLOAD_SHARED_SECRET``.  Then on your client, you generate your signature by
-concatenating the correspondent, title, and the secret, and then using sha256
-to generate a hexdigest.
-
-If you're using Python, this is what that looks like:
 
 .. code:: python
 
+    import os
+
     from hashlib import sha256
-    signature = sha256(correspondent + title + secret).hexdigest()
+
+    import requests
+    from requests.auth import HTTPBasicAuth
+
+    # You authenticate via BasicAuth or with a session id.
+    # We use BasicAuth here
+    username = "my-username"
+    password = "my-super-secret-password"
+
+    # Where you have Paperless installed and listening
+    url = "http://localhost:8000/push"
+
+    # Document metadata
+    correspondent = "Test Correspondent"
+    title = "Test Title"
+
+    # The local file you want to push
+    path = "/path/to/some/directory/my-document.pdf"
+
+
+    with open(path, "rb") as f:
+
+        response = requests.post(
+            url=url,
+            data={"title": title,  "correspondent": correspondent},
+            files={"document": (os.path.basename(path), f, "application/pdf")},
+            auth=HTTPBasicAuth(username, password),
+            allow_redirects=False
+        )
+
+        if response.status_code == 202:
+
+            # Everything worked out ok
+            print("Upload successful")
+
+        else:
+
+            # If you don't get a 202, it's probably because your credentials
+            # are wrong or something.  This will give you a rough idea of what
+            # happened.
+
+            print("We got HTTP status code: {}".format(response.status_code))
+            for k, v in response.headers.items():
+                print("{}: {}".format(k, v))

paperless.conf.example

@@ -5,7 +5,7 @@
 
 
 ###############################################################################
-####                       Paths  and folders                              ####
+####                         Paths & Folders                               ####
 ###############################################################################
 
 # This where your documents should go to be consumed.  Make sure that it exists
@@ -39,7 +39,11 @@ PAPERLESS_CONSUME_MAIL_PASS=""
 
 # Override the default IMAP inbox here. If not set Paperless defaults to
 # "INBOX".
-#PAPERLESS_CONSUME_MAIL_INBOX=""
+#PAPERLESS_CONSUME_MAIL_INBOX="INBOX"
+
+# Any email sent to the target account that does not contain this text will be
+# ignored.
+PAPERLESS_EMAIL_SECRET=""
 
 
 ###############################################################################
@@ -61,11 +65,6 @@ PAPERLESS_CONSUME_MAIL_PASS=""
 PAPERLESS_PASSPHRASE="secret"
 
 
-# If you intend to consume documents either via HTTP POST or by email, you must
-# have a shared secret here.
-PAPERLESS_SHARED_SECRET=""
-
-
 # The secret key has a default that should be fine so long as you're hosting
 # Paperless on a closed network.  However, if you're putting this anywhere
 # public, you should change the key to something unique and verbose.

scripts/docker-entrypoint.sh

@@ -7,9 +7,9 @@ map_uidgid() {
     USERMAP_ORIG_UID=$(id -g paperless)
     USERMAP_GID=${USERMAP_GID:-${USERMAP_UID:-$USERMAP_ORIG_GID}}
     USERMAP_UID=${USERMAP_UID:-$USERMAP_ORIG_UID}
-    if [[ ${USERMAP_UID} != ${USERMAP_ORIG_UID} || ${USERMAP_GID} != ${USERMAP_ORIG_GID} ]]; then
+    if [[ ${USERMAP_UID} != "${USERMAP_ORIG_UID}" || ${USERMAP_GID} != "${USERMAP_ORIG_GID}" ]]; then
         echo "Mapping UID and GID for paperless:paperless to $USERMAP_UID:$USERMAP_GID"
-        groupmod -g ${USERMAP_GID} paperless
+        groupmod -g "${USERMAP_GID}" paperless
         sed -i -e "s|:${USERMAP_ORIG_UID}:${USERMAP_GID}:|:${USERMAP_UID}:${USERMAP_GID}:|" /etc/passwd
     fi
 }
@@ -62,11 +62,11 @@ install_languages() {
     # Loop over languages to be installed
     for lang in "${langs[@]}"; do
         pkg="tesseract-ocr-$lang"
-        if dpkg -s "$pkg" 2>&1 > /dev/null; then
+        if dpkg -s "$pkg" > /dev/null 2>&1; then
             continue
         fi
 
-        if ! apt-cache show "$pkg" 2>&1 > /dev/null; then
+        if ! apt-cache show "$pkg" > /dev/null 2>&1; then
             continue
         fi
 

src/documents/forms.py

@@ -2,7 +2,6 @@ import magic
 import os
 
 from datetime import datetime
-from hashlib import sha256
 from time import mktime
 
 from django import forms
@@ -14,7 +13,6 @@ from .consumer import Consumer
 
 class UploadForm(forms.Form):
 
-    SECRET = settings.SHARED_SECRET
     TYPE_LOOKUP = {
         "application/pdf": Document.TYPE_PDF,
         "image/png": Document.TYPE_PNG,
@@ -32,10 +30,9 @@ class UploadForm(forms.Form):
         required=False
     )
     document = forms.FileField()
-    signature = forms.CharField(max_length=256)
 
     def __init__(self, *args, **kwargs):
-        forms.Form.__init__(*args, **kwargs)
+        forms.Form.__init__(self, *args, **kwargs)
         self._file_type = None
 
     def clean_correspondent(self):
@@ -82,17 +79,6 @@ class UploadForm(forms.Form):
 
         return document
 
-    def clean(self):
-
-        corresp = self.cleaned_data.get("correspondent")
-        title = self.cleaned_data.get("title")
-        signature = self.cleaned_data.get("signature")
-
-        if sha256(corresp + title + self.SECRET).hexdigest() == signature:
-            return self.cleaned_data
-
-        raise forms.ValidationError("The signature provided did not validate")
-
     def save(self):
         """
         Since the consumer already does a lot of work, it's easier just to save
@@ -104,7 +90,7 @@ class UploadForm(forms.Form):
         title = self.cleaned_data.get("title")
         document = self.cleaned_data.get("document")
 
-        t = int(mktime(datetime.now()))
+        t = int(mktime(datetime.now().timetuple()))
         file_name = os.path.join(
             Consumer.CONSUME,
             "{} - {}.{}".format(correspondent, title, self._file_type)

src/documents/mail.py

@@ -43,7 +43,10 @@ class Message(Loggable):
     and n attachments, and that we don't care about the message body.
     """
 
-    SECRET = settings.SHARED_SECRET
+    SECRET = os.getenv(
+        "PAPERLESS_EMAIL_SECRET",
+        os.getenv("PAPERLESS_SHARED_SECRET")  # TODO: Remove after 2017/09
+    )
 
     def __init__(self, data, group=None):
         """
@@ -153,15 +156,16 @@ class MailFetcher(Loggable):
         Loggable.__init__(self)
 
         self._connection = None
-        self._host = settings.MAIL_CONSUMPTION["HOST"]
+        self._host = os.getenv("PAPERLESS_CONSUME_MAIL_HOST")
-        self._port = settings.MAIL_CONSUMPTION["PORT"]
+        self._port = os.getenv("PAPERLESS_CONSUME_MAIL_PORT")
-        self._username = settings.MAIL_CONSUMPTION["USERNAME"]
+        self._username = os.getenv("PAPERLESS_CONSUME_MAIL_USER")
-        self._password = settings.MAIL_CONSUMPTION["PASSWORD"]
+        self._password = os.getenv("PAPERLESS_CONSUME_MAIL_PASS")
-        self._inbox = settings.MAIL_CONSUMPTION["INBOX"]
+        self._inbox = os.getenv("PAPERLESS_CONSUME_MAIL_INBOX", "INBOX")
 
         self._enabled = bool(self._host)
 
         self.last_checked = datetime.datetime.now()
+        print(self._connection, self._host, self._port, self._username, self._password, self._inbox, self._enabled, self.last_checked)
 
     def pull(self):
         """

src/documents/views.py

@@ -1,5 +1,4 @@
-from django.http import HttpResponse
+from django.http import HttpResponse, HttpResponseBadRequest
-from django.views.decorators.csrf import csrf_exempt
 from django.views.generic import DetailView, FormView, TemplateView
 from django_filters.rest_framework import DjangoFilterBackend
 from paperless.db import GnuPG
@@ -81,15 +80,12 @@ class PushView(SessionOrBasicAuthMixin, FormView):
 
     form_class = UploadForm
 
-    @classmethod
-    def as_view(cls, **kwargs):
-        return csrf_exempt(FormView.as_view(**kwargs))
-
     def form_valid(self, form):
-        return HttpResponse("1")
+        form.save()
+        return HttpResponse("1", status=202)
 
     def form_invalid(self, form):
-        return HttpResponse("0")
+        return HttpResponseBadRequest(str(form.errors))
 
 
 class CorrespondentViewSet(ModelViewSet):

src/paperless/checks.py

@@ -84,3 +84,20 @@ def binaries_check(app_configs, **kwargs):
             check_messages.append(Warning(error.format(binary), hint))
 
     return check_messages
+
+
+@register()
+def config_check(app_configs, **kwargs):
+    warning = (
+        "It looks like you have PAPERLESS_SHARED_SECRET defined.  Note that "
+        "in the \npast, this variable was used for both API authentication "
+        "and as the mail \nkeyword.  As the API no no longer uses it, this "
+        "variable has been renamed to \nPAPERLESS_EMAIL_SECRET, so if you're "
+        "using the mail feature, you'd best update \nyour variable name.\n\n"
+        "The old variable will stop working in a few months."
+    )
+
+    if os.getenv("PAPERLESS_SHARED_SECRET"):
+        return [Warning(warning)]
+
+    return []

src/paperless/settings.py

@@ -69,6 +69,7 @@ INSTALLED_APPS = [
 
     "rest_framework",
     "crispy_forms",
+    "django_filters"
 
 ]
 
@@ -236,20 +237,6 @@ CONSUMPTION_DIR = os.getenv("PAPERLESS_CONSUMPTION_DIR")
 # slowly, you may want to use a higher value than the default.
 CONSUMER_LOOP_TIME = int(os.getenv("PAPERLESS_CONSUMER_LOOP_TIME", 10))
 
-# If you want to use IMAP mail consumption, populate this with useful values.
-# If you leave HOST set to None, we assume you're not going to use this
-# feature.
-MAIL_CONSUMPTION = {
-    "HOST": os.getenv("PAPERLESS_CONSUME_MAIL_HOST"),
-    "PORT": os.getenv("PAPERLESS_CONSUME_MAIL_PORT"),
-    "USERNAME": os.getenv("PAPERLESS_CONSUME_MAIL_USER"),
-    "PASSWORD": os.getenv("PAPERLESS_CONSUME_MAIL_PASS"),
-    # If True, use SSL/TLS to connect
-    "USE_SSL": os.getenv("PAPERLESS_CONSUME_MAIL_USE_SSL", "y").lower() == "y",
-    # The name of the inbox on the server
-    "INBOX": os.getenv("PAPERLESS_CONSUME_MAIL_INBOX", "INBOX")
-}
-
 # This is used to encrypt the original documents and decrypt them later when
 # you want to download them.  Set it and change the permissions on this file to
 # 0600, or set it to `None` and you'll be prompted for the passphrase at
@@ -259,11 +246,6 @@ MAIL_CONSUMPTION = {
 # files.
 PASSPHRASE = os.getenv("PAPERLESS_PASSPHRASE")
 
-# If you intend to use the "API" to push files into the consumer, you'll need
-# to provide a shared secret here.  Leaving this as the default will disable
-# the API.
-SHARED_SECRET = os.getenv("PAPERLESS_SHARED_SECRET", "")
-
 # Trigger a script after every successful document consumption?
 PRE_CONSUME_SCRIPT = os.getenv("PAPERLESS_PRE_CONSUME_SCRIPT")
 POST_CONSUME_SCRIPT = os.getenv("PAPERLESS_POST_CONSUME_SCRIPT")

src/paperless/urls.py

@@ -1,27 +1,12 @@
-"""paperless URL Configuration
-
-The `urlpatterns` list routes URLs to views. For more information please see:
-    https://docs.djangoproject.com/en/1.10/topics/http/urls/
-Examples:
-Function views
-    1. Add an import:  from my_app import views
-    2. Add a URL to urlpatterns:  url(r'^$', views.home, name='home')
-Class-based views
-    1. Add an import:  from other_app.views import Home
-    2. Add a URL to urlpatterns:  url(r'^$', Home.as_view(), name='home')
-Including another URLconf
-    1. Add an import:  from blog import urls as blog_urls
-    2. Import the include() function: from django.conf.urls import url, include
-    3. Add a URL to urlpatterns:  url(r'^blog/', include(blog_urls))
-"""
 from django.conf import settings
 from django.conf.urls import url, static, include
 from django.contrib import admin
+from django.views.decorators.csrf import csrf_exempt
 
 from rest_framework.routers import DefaultRouter
 
 from documents.views import (
-    IndexView, FetchView, PushView,
+    FetchView, PushView,
     CorrespondentViewSet, TagViewSet, DocumentViewSet, LogViewSet
 )
 from reminders.views import ReminderViewSet
@@ -42,9 +27,6 @@ urlpatterns = [
     ),
     url(r"^api/", include(router.urls, namespace="drf")),
 
-    # Normal pages (coming soon)
-    # url(r"^$", IndexView.as_view(), name="index"),
-
     # File downloads
     url(
         r"^fetch/(?P<kind>doc|thumb)/(?P<pk>\d+)$",
@@ -52,15 +34,15 @@ urlpatterns = [
         name="fetch"
     ),
 
+    # File uploads
+    url(r"^push$", csrf_exempt(PushView.as_view()), name="push"),
+
     # The Django admin
     url(r"admin/", admin.site.urls),
     url(r"", admin.site.urls),  # This is going away
 
 ] + static.static(settings.MEDIA_URL, document_root=settings.MEDIA_ROOT)
 
-if settings.SHARED_SECRET:
-    urlpatterns.insert(0, url(r"^push$", PushView.as_view(), name="push"))
-
 # Text in each page's <h1> (and above login form).
 admin.site.site_header = 'Paperless'
 # Text at the end of each page's <title>.

src/paperless/version.py

@@ -1 +1 @@
-__version__ = (0, 5, 0)
+__version__ = (0, 6, 0)