Merge branch 'improve-randomness' of framasky/masto-image-bot into master

My bot @gracyimp@botsin.space gave me the same image twice this morning,
within a few seconds, so I asked myself how to improve the randomness.
Indeed, this allowed my bot to publish some images that never were
published!

.gitignore

@@ -4,3 +4,5 @@ config.txt
 __pycache__/*
 TootHTMLParser.pyc
 blacklist.json
+collection.json
+output.jpg

README.md

@@ -1,39 +1,57 @@
 # masto-image-bot
 
-Un bot qui récupère une image random en local et la publie
+A bot that fetches a random local image and publish it on Mastodon.
 
-Copiez le fichier `config.sample.txt` en `config.txt`, ajoutez le chemin de votre dossier image.
-Remplissez le fichier `secrets/secrets.txt` et remplissez le avec les code que vous trouverez dans l'onglet développeur de votre compte Mastodon.
+Copy the file `config.sample.txt` to `config.txt` and add the path to your images directory.
+File the file `secrets/secrets.txt` with the codes you will find in the developper tab of your Mastodon account.
+You can also register your bot on a Mastodon instance and get the needed codes with the help of the script [register-app](https://framagit.org/fiat-tux/hat-softwares/mastodon/register-app).
 
-## Configure it
+## Install the dependencies
+
+```
+pip3 install -r requirements.txt
+```
+
+## Configure the bot
 
 Copy `config.sample.txt` to `config.txt` and replace data by your data.
 
 If you don't want any "spoiler text", just leave the line empty.
 
-| **Field name** | Description                                                            | Value                  |
-| -------------- | ---------------------------------------------------------------------- | ---------------------- |
-| img_path       | Path where image are located                                           | /home/bot/img | ../bot |
-| name           | Name of your bot (name after @). Is usefull to hide it's name in reply | mybot                  |
-| sensitive      | Hide picture behind "sensitive content" mask or not                    | yes | no               |
-| default_text   | Text for --img option                                                  | string                 |
-| spoiler_text   | Text for every spoiler (or CW) (for --img or --stream option)          | string                 |
-| limit          | Limit send per minute per person                                       | int                    |
-| limit_hour     | Limit send par hour per person                                         | int                    |
+| **Field name**      | Description                                                                                                                 | Value                  |
+| ------------------- | --------------------------------------------------------------------------------------------------------------------------- | ---------------------- |
+| img_path            | Path where image are located                                                                                                | /home/bot/img | ../bot |
+| name                | Name of your bot (name after @). Is usefull to hide it's name in reply                                                      | mybot                  |
+| secrets_filepath    | Path where located secret                                                                                                   | secrets/secrets.txt    |
+| log_filepath        | Path where located log file                                                                                                 | activity.log           |
+| blacklist_filepath  | Path where located blacklist file                                                                                           | blacklist.json         |
+| collection_filepath | Path where located distant image collection                                                                                 | collection.json        |
+| sensitive           | Hide picture behind "sensitive content" mask or not                                                                         | yes | no               |
+| default_text        | Text for --img option                                                                                                       | string                 |
+| spoiler_text        | Text for every spoiler (or CW) (for --img or --stream option)                                                               | string                 |
+| limit               | Limit send per minute per person                                                                                            | int                    |
+| limit_hour          | Limit send par hour per person                                                                                              | int                    |
+| collection_url      | URL of website you deserve image. `<collection>` is a variable who depend on collection.json (you can remove this variable) | string                 |
+| unsplash_client_id  | Access key of your Unsplash App (you can create it on api.unsplash.com )                                                    | string                 |
 
 Copy `blacklist.sample.json` to `blacklist.json` and replace or add accounts that should not receive any image
 
+Copy `collection.sample.json` to `collection.json` and add collection for your bot
+
 ## Use it
 
 ```
-usage: bot.py [-h] [-i] [-s]
+usage: bot.py [-h] [-i] [-s SOURCE] [--stream]
 
 Choose between image or streaming
 
 optional arguments:
-  -h, --help    show this help message and exit
-  -i, --img     post image
-  -s, --stream  stream user profile
+  -h, --help            show this help message and exit
+  -i, --img             post image
+  -s SOURCE, --source SOURCE
+                        Source of image [ local | distant | unsplash-random ]
+  --stream              stream user profile
+
 ```
 
 `--img` option send image.
@@ -55,5 +73,15 @@ Type=simple
 User=masto-bot
 TimeoutSec=15
 WorkingDirectory=/home/masto-bot/
-ExecStart=/usr/bin/python3 bot.py --stream
+ExecStart=/usr/bin/python3 bot.py --stream --source=local
+
+[Install]
+WantedBy=multi-user.target
+```
+
+Then do
+```
+systemctl daemon-reload
+systemctl enable bot.service
+systemctl start bot.service
 ```

TootHTMLParser.py

@@ -1,20 +0,0 @@
-from html.parser import HTMLParser
-
-class TootHTMLParser(HTMLParser):
-    def __init__(self):
-        super().__init__()
-        self.txt = ""
-    def handle_data(self, data):
-        self.txt += str(data).lstrip().rstrip().lower() + " "
-#
-#
-# content = ""
-# with open("input") as f:
-#     content = f.readlines()
-# content = set(content)
-# parser = TootHTMLParser()
-# for word in content:
-#     parser.feed(word)
-# with open("output", "w+") as f:
-#     f.write(parser.txt)
-

bot.py

@@ -7,171 +7,268 @@ from mastodon import StreamListener
 from lxml import html
 from logging.handlers import RotatingFileHandler
 from pprint import pprint
+from random import randint
 from utils.config import get_parameter, init_log, init_mastodon
+from PIL import Image
+from io import BytesIO
 
-import requests, os, random, sys, time, json, logging, argparse, re
-
-config_file        = "config.txt"
-secrets_filepath   = "secrets/secrets.txt"
-log_filepath       = "activity.log"
-blacklist_filepath = "blacklist.json"
-log = init_log(log_filepath)
-mastodon = init_mastodon(config_file, secrets_filepath)
+import requests, os, random, sys, time, json, logging, argparse, re, shutil
+config_file         = "config.txt"
+secrets_filepath    = get_parameter("secrets_filepath", config_file)
+log_filepath        = get_parameter("log_filepath", config_file)
+blacklist_filepath  = get_parameter("blacklist_filepath", config_file)
+collection_filepath = get_parameter("collection_filepath", config_file)
+log                 = init_log(log_filepath)
+mastodon            = init_mastodon(config_file, secrets_filepath)
 
 blacklist_file = open(blacklist_filepath,'r')
-BLACKLIST = json.loads(blacklist_file.read())
+BLACKLIST      = json.loads(blacklist_file.read())
 blacklist_file.close()
 
+mime_dict = {'.jpg': 'image/jpeg', '.jpe': 'image/jpeg', '.jpeg': 'image/jpeg', '.png': 'image/png', '.gif': 'image/gif'}
 
-def post_img(mastodon, text, log, config):
+def post_img_local(mastodon, text, log, config):
     img_path = get_parameter("img_path", config)
-    file = random.choice(os.listdir(img_path+"/")) 
-    image_byte = open(img_path+"/"+file, "rb").read()
 
-    file, ext = os.path.splitext(file)
-    mime_dict = {'.jpg': 'image/jpeg', '.jpe': 'image/jpeg', '.jpeg': 'image/jpeg', '.png': 'image/png', '.gif': 'image/gif'}
-    
-    mime = mime_dict[str.lower(ext)]
-    #try:
+    continu = True;
+    while continu:
+        secure_random = random.SystemRandom()
+        file = secure_random.choice(os.listdir(img_path+"/"))
+        if os.path.isdir(img_path+file):
+            img_path = img_path+file+"/"
+        else:
+            if ".zip" not in file:
+                continu = False
 
-    #except KeyError:
-    #    mime = None;
-    #    pass
+    im            = Image.open(img_path+ file)
+    width, height = im.size
+    NEW_WIDTH = 2048
+    if width > 2048:
+        difference_percent = NEW_WIDTH / width
+        new_height         = height * difference_percent
+        size               = new_height, NEW_WIDTH
+        im                 = im.resize((int(NEW_WIDTH), int(new_height)))
+        im.save('resize_img.jpg')
+        file = "resize_img.jpg"
+        shutil.copyfile(file, "/tmp/"+file)
+    else:
+        log.debug("no resize")
+        shutil.copyfile(img_path+file, "/tmp/"+file)
+
+    image_byte = open("/tmp/"+file, "rb").read()
+    file, ext  = os.path.splitext(file)
+    os.remove("/tmp/"+file+ext)
+
+    try:
+        mime = mime_dict[str.lower(ext)]
+    except KeyError:
+        mime = None;
+        log.error(ext + " is not present on mime_dict, please add this")
+        pass
 
     media_dict = mastodon.media_post(image_byte, mime)
     return media_dict;
-    
+
+def post_unsplash_random_image(mastodon, log, config):
+    collection_url = get_parameter("collection_url", config)
+    unsplash_client_id  = get_parameter("unsplash_client_id", config)
+    collecion_file = open(collection_filepath,'r')
+    collections    = json.loads(collecion_file.read())
+    collecion_file.close()
+
+    count_collection = len(collections)-1
+    if count_collection > -1:
+      id_collection    = randint(0,count_collection)
+      collection_url="&collections="+str(collections[id_collection])
+    else:
+      collection_url=''
+    response    = requests.get("https://api.unsplash.com/photos/random?client_id="+unsplash_client_id+collection_url)
+    randim_json = json.loads(response.text)
+    randim_url  = "{}&q=85&crop=entropy&cs=tinysrgb&w=2048&fit=max".format(randim_json['urls']['raw'])
+
+    img_response = requests.get(randim_url)
+    pattern      = Image.open(BytesIO(img_response.content), "r").convert('RGB')
+    pattern.save('output.jpg')
+
+    media_dict = mastodon.media_post("output.jpg")
+
+    toot = "Shot by {} ({})\n{}".format(randim_json['user']['name'], randim_json['user']['links']['html'], randim_json['links']['html'])
+
+    return { 'media_dict': media_dict, 'toot': toot };
+
+def post_img_distant(mastodon, text, log, config):
+
+    collection_url = get_parameter("collection_url", config)
+    collecion_file = open(collection_filepath,'r')
+    collections    = json.loads(collecion_file.read())
+    collecion_file.close()
+
+    count_collection = len(collections)-1
+    id_collection    = randint(0,count_collection)
+
+
+    collection_url = collection_url.replace("<collection>", str(collections[id_collection]))
+    response       = requests.get(collection_url)
+    pattern        = Image.open(BytesIO(response.content), "r").convert('RGB')
+    pattern.save('output.jpg')
+
+    media_dict = mastodon.media_post("output.jpg")
+    return media_dict;
+
+cleanr = re.compile('<.*?>')
 def cleanhtml(raw_html):
-    cleanr = re.compile('<.*?>')
     cleantext = re.sub(cleanr, '', raw_html)
     return cleantext
 
 class BotListener(StreamListener):
 
-	# use only notification
-	def on_notification(self, notification):
+    def __init__(self, args):
+        self.args = args
 
-			# catch only mention in notification
-			if notification['type'] == 'mention':
-				log.debug("Got a mention")
-				if notification["account"]["bot"] == False:
-					sender = notification['account']['acct'] # Get sender name
-					if sender in BLACKLIST:
-						log.info("Service refused to %s" % sender)
-						return
-					sender_hour_filename = "limiter/hour/" + sender; # Forge file for limiter
-					sender_minute_filename = "limiter/minute/" + sender; # Forge file for limiter
-					
-					if os.path.isfile(sender_hour_filename): # Check if file exist
-						log.debug("Sender file exist")
-						statbuf = os.stat(sender_hour_filename) 
-						last_edit = int(statbuf.st_mtime)
-						ts = int(time.time())
-						if ts - last_edit > 3599: # check if file is modified 1 hour after last edition
-							log.debug("file is too old")
-							f = open(sender_hour_filename,'w')
-							f.write(str(1)) # reset counter
-							f.close()
-							can_continue = True
-						else: 
-							log.debug("file is young")
-							f = open(sender_hour_filename,'r+')
-							limit = int(get_parameter("limit_hour", config_file))
-							number_of_mention = int(f.read())
-							if number_of_mention < limit: # limit of mention per hour is limit_hour
-								log.debug("Sender have less of limit requests")
-								f.seek(0)
-								f.write(str(number_of_mention + 1))
-								can_continue = True
-							else:
-								log.debug("Sender have more of limit requests")
-								can_continue = False # if number of mention is for, user can't receive anything
-							f.close()
-					else: # File not exist, create it and initialise it
-						log.debug("Sender file not exist")
-						f = open(sender_hour_filename,"w+")
-						f.write(str(1))
-						f.close()
-						can_continue = True
-					
-					if can_continue:
-						if os.path.isfile(sender_minute_filename): # Check if file exist
-							log.debug("Sender file exist")
-							statbuf = os.stat(sender_minute_filename) 
-							last_edit = int(statbuf.st_mtime)
-							ts = int(time.time())
-							if ts - last_edit > 59: # check if file is modified 1 minute after last edition
-								log.debug("file is too old")
-								f = open(sender_minute_filename,'w')
-								f.write(str(1)) # reset counter
-								f.close()
-								can_continue = True
-							else: 
-								log.debug("file is young")
-								f = open(sender_minute_filename,'r+')
-								limit = int(get_parameter("limit", config_file))
-								number_of_mention = int(f.read())
-								if number_of_mention < limit: # limit of mention per minute is 4
-									log.debug("Sender have less of limit requests")
-									f.seek(0)
-									f.write(str(number_of_mention + 1))
-									can_continue = True
-								else:
-									log.debug("Sender have more of limit requests")
-									can_continue = False # if number of mention is for, user can't receive anything
-									file = open(sender_hour_filename,'r+')
-									number_of_mention = int(file.read())
-									file.seek(0)
-									file.write(str(number_of_mention - 1))
-									file.close()
-								f.close()
-						else: # File not exist, create it and initialise it
-							log.debug("Sender file not exist")
-							f = open(sender_minute_filename,"w+")
-							f.write(str(1))
-							f.close()
-							can_continue = True
+    # use only notification
+    def on_notification(self, notification):
 
-					if can_continue:
-						id = notification['status']['id']
-						visibility = notification['status']['visibility']
-						if visibility == 'public':
-							visibility = 'unlisted'
-						mentions = notification['status']['mentions']
-						text = "@" + notification['status']["account"]["acct"] + " "
-						for mention in mentions:
-							if mention["acct"] != get_parameter("name_bot", config_file):
-								text = text + "@" + mention["acct"] + " "
-			    		
-						if get_parameter("sensitive", config_file) == "yes":
-							sensitive = True
-						else:
-							sensitive = False
-						media_dict = post_img(mastodon, get_parameter("default_text", config_file), log, config_file)
-						mastodon.status_post(text, None, media_ids=[media_dict], sensitive=sensitive, visibility=visibility, spoiler_text=get_parameter("spoiler_text", config_file))
-					else: 
-						log.debug("no picture send :(")
-						pass
-			else:
-				log.debug("Nevermind")
+        # catch only mention in notification
+        if notification['type'] == 'mention':
+            log.debug("Got a mention")
+            if notification["account"]["bot"] == False:
+                sender = notification['account']['acct'] # Get sender name
+                if sender in BLACKLIST:
+                    log.info("Service refused to %s" % sender)
+                    return
+                sender_hour_filename   = "limiter/hour/" + sender; # Forge file for limiter
+                sender_minute_filename = "limiter/minute/" + sender; # Forge file for limiter
+
+                if os.path.isfile(sender_hour_filename): # Check if file exist
+                    log.debug("Sender file exist")
+                    statbuf   = os.stat(sender_hour_filename)
+                    last_edit = int(statbuf.st_mtime)
+                    ts        = int(time.time())
+                    if ts - last_edit > 3599: # check if file is modified 1 hour after last edition
+                        log.debug("file is too old")
+                        f = open(sender_hour_filename,'w')
+                        f.write(str(1)) # reset counter
+                        f.close()
+                        can_continue = True
+                    else:
+                        log.debug("file is young")
+                        f     = open(sender_hour_filename,'r+')
+                        limit = int(get_parameter("limit_hour", config_file))
+                        number_of_mention = int(f.read())
+                        if number_of_mention < limit: # limit of mention per hour is limit_hour
+                            log.debug("Sender have less of limit requests")
+                            f.seek(0)
+                            f.write(str(number_of_mention + 1))
+                            can_continue = True
+                        else:
+                            log.debug("Sender have more of limit requests")
+                            can_continue = False # if number of mention is for, user can't receive anything
+                        f.close()
+                else: # File not exist, create it and initialise it
+                    log.debug("Sender file not exist")
+                    f = open(sender_hour_filename,"w+")
+                    f.write(str(1))
+                    f.close()
+                    can_continue = True
+
+                if can_continue:
+                    if os.path.isfile(sender_minute_filename): # Check if file exist
+                        log.debug("Sender file exist")
+                        statbuf   = os.stat(sender_minute_filename)
+                        last_edit = int(statbuf.st_mtime)
+                        ts        = int(time.time())
+                        if ts - last_edit > 59: # check if file is modified 1 minute after last edition
+                            log.debug("file is too old")
+                            f = open(sender_minute_filename,'w')
+                            f.write(str(1)) # reset counter
+                            f.close()
+                            can_continue = True
+                        else:
+                            log.debug("file is young")
+                            f                 = open(sender_minute_filename,'r+')
+                            limit             = int(get_parameter("limit", config_file))
+                            number_of_mention = int(f.read())
+                            if number_of_mention < limit: # limit of mention per minute is 4
+                                log.debug("Sender have less of limit requests")
+                                f.seek(0)
+                                f.write(str(number_of_mention + 1))
+                                can_continue = True
+                            else:
+                                log.debug("Sender have more of limit requests")
+                                can_continue      = False # if number of mention is for, user can't receive anything
+                                file              = open(sender_hour_filename,'r+')
+                                number_of_mention = int(file.read())
+                                file.seek(0)
+                                file.write(str(number_of_mention - 1))
+                                file.close()
+                            f.close()
+                    else: # File not exist, create it and initialise it
+                        log.debug("Sender file not exist")
+                        f = open(sender_minute_filename,"w+")
+                        f.write(str(1))
+                        f.close()
+                        can_continue = True
+
+                if can_continue:
+                    id         = notification['status']['id']
+                    visibility = notification['status']['visibility']
+                    if visibility == 'public':
+                        visibility = 'unlisted'
+                    mentions = notification['status']['mentions']
+                    text     = "@" + notification['status']["account"]["acct"] + " "
+                    for mention in mentions:
+                        if mention["acct"] != get_parameter("name_bot", config_file):
+                            text = text + "@" + mention["acct"] + " "
+
+                    if get_parameter("sensitive", config_file) == "yes":
+                        sensitive = True
+                    else:
+                        sensitive = False
+
+                    if self.args.source == "local":
+                        media_dict = post_img_local(mastodon, get_parameter("default_text", config_file), log, config_file)
+                    elif self.args.source == "distant":
+                        media_dict = post_img_distant(mastodon, get_parameter("default_text", config_file), log, config_file)
+                    elif self.args.source == "unsplash-random":
+                        resp       = post_unsplash_random_image(mastodon, log, config_file)
+                        text       = text + "\n" + resp['toot']
+                        media_dict = resp['media_dict']
+
+                    mastodon.status_post(text, id, media_ids=[media_dict], sensitive=sensitive, visibility=visibility, spoiler_text=get_parameter("spoiler_text", config_file))
+                else:
+                    log.debug("no picture send :(")
+                    pass
+        else:
+            log.debug("Nevermind")
 
 
 def main():
-    
+
     parser = argparse.ArgumentParser(description='Choose between image or streaming')
     parser.add_argument("-i", "--img", action='store_true', help="post image")
-    parser.add_argument("-s", "--stream", action="store_true", help="stream user profile")
+    parser.add_argument("-s", "--source", help="Source of image [ local | distant | unsplash-random ]")
+    parser.add_argument("--stream", action="store_true", help="stream user profile")
     args = parser.parse_args()
     if args.img:
-        media_dict = post_img(mastodon, get_parameter("default_text", config_file), log, config_file)
+        text = get_parameter("default_text", config_file)
+        if args.source == "local":
+            media_dict = post_img_local(mastodon, get_parameter("default_text", config_file), log, config_file)
+        elif args.source == "distant":
+            media_dict = post_img_distant(mastodon, get_parameter("default_text", config_file), log, config_file)
+        elif args.source == "unsplash-random":
+            resp       = post_unsplash_random_image(mastodon, log, config_file)
+            text       = resp['toot']
+            media_dict = resp['media_dict']
+
         if get_parameter("sensitive", config_file) == "yes":
-        	sensitive = True
+            sensitive = True
         else:
-        	sensitive = False
-        mastodon.status_post(get_parameter("default_text", config_file), None, media_ids=[media_dict], sensitive=sensitive, visibility='public', spoiler_text=get_parameter("spoiler_text", config_file))
+            sensitive = False
+        mastodon.status_post(text, None, media_ids=[media_dict], sensitive=sensitive, visibility='public', spoiler_text=get_parameter("spoiler_text", config_file))
         sys.exit()
+
     elif args.stream:
-        stream = BotListener();
+        stream = BotListener(args);
         while True:
             try:
                 log.info("Start listening...")

collection.sample.json

@@ -0,0 +1 @@
+[]

config.sample.txt

@@ -1,6 +1,13 @@
 img_path: uri_path
 name_bot: bot
+secrets_filepath: secrets/secrets.txt
+log_filepath: activity.log
+blacklist_filepath: blacklist.json
+collection_filepath: collection.json
 sensitive: False
 default_text: some text here
 spoiler_text: some text here
 limit: 2
+limit_hour: 10
+collection_url: https://source.unsplash.com/collection/<collection>/
+unsplash_client_id: 03ad5bfbaa0acd6c96a728d425e533683ec25e5fb7fcf99f6461720b3d0d75a1

rename_all_file.sh

@@ -0,0 +1,28 @@
+#!/bin/bash
+
+function generate_random_char {
+	echo $( dd if=/dev/urandom bs=16 count=1|base64) > /tmp/rename_all_image
+	cp /tmp/rename_all_image /tmp/rename_all_image.back
+	sed -ie 's/[!@#\+\/$%^&*()=]//g' /tmp/rename_all_image.back
+	NEW_FILENAME=$(cat /tmp/rename_all_image.back)
+	EXTENSION=$(echo $img | cut -f 2 -d '.')
+	echo $NEW_FILENAME"."$EXTENSION
+}
+
+function move_file {
+	NEW_FILE=$(generate_random_char)
+	filepath=$2
+	IMG=$1
+	#echo $filepath"/"$NEW_FILE
+	if [ ! -f $filepath"/"$NEW_FILE ]; then
+		#mv $IMG $NEW_FILE
+		mv $IMG $filepath"/"$NEW_FILE
+	else
+		move_file $IMG
+	fi
+}
+
+for img in `ls $1/*`; do
+	filepath=$1
+	move_file $img $filepath
+done

requirements.txt

@@ -1,3 +1,4 @@
 Mastodon.py
 lxml
 requests
+Pillow