Artificial intelligence tweeting Bird Feeder

Create a bird watching Artificial Intelligence that runs on a raspberry and tweets short videos every time it detects a bird. You can find mine at https://twitter.com/birdfeederAI

1. Get the hardware

To build this bird watcher you will need the following hardware parts. Total cost EUR 255

(optional) PoE switch (link) cost EUR 118
(optional) PoE hat for Raspberry Pi (link) cost EUR 28.99
Raspberry pi 4 8 GB (link) cost EUR 81.66
Raspberry pi camera (link) cost EUR 33.90
64GB SD card (link) cost EUR 11.99 (update, since raspberry pi 3B you can boot from USB which is faster and fails less, so better get a 64GB usb 3.1 pendrive (link) 11.63 EUR )

2. Set up base infrastructure

Install raspberry pi OS (link)
Activate the camera (link)
Log into the raspberry pi and create a python virtual environment with base libraries
1. mkdir -p /home/pi/dev/birdfeederAI
2. cd /home/pi/dev/birdfeederAI
3. python3 -m venv ./venv
  1. This creates the python virtual environment.
4. source ./venv/bin/activate
  1. This activates the python virtual environment. All modules installed by pip will be installed only for this virtual environment
5. which python
  1. Now that we have created and activated our virtual environment this should return: “/home/pi/dev/birdfeederAI/venv/bin/python”
6. which pip
  1. Now that we have created and activated our virtual environment this should return: “/home/pi/dev/birdfeederAI/venv/bin/pip”
7. which pip3
  1. Now that we have created and activated our virtual environment this should return: “/home/pi/dev/birdfeederAI/venv/bin/pip3”
8. pip list
  1. this will list the python modules we have installed for our virtual environment.
9. pip install --upgrade pip
  1. This will install the latest version of pip
10. pip install opencv-python twython
  1. install the opencv python module to process video, twython to send tweets
Fix all the missing libs
1. sudo apt install apt-file
2. sudo apt-file update
3. for i in find /home/pi/dev/birdfeederAI/venv/lib |grep so$|xargs ldd|grep "not found"|awk '{print $1;}'; do apt-file search $i|awk 'BEGIN{FS=":"};{print $1;}'; done|sort|uniq|xargs apt install
  1. This will list the libraries which are not installed and install them for you. If something went wrong look here (link)
Download the pre-trained model MobileNet-SSD
1. cd /home/pi/dev/birdfeederAI
1. git clone https://github.com/chuanqi305/MobileNet-SSD.git
Check that the base infrastructure is correctly installed
1. raspistill -o mypicture.jpg
  1. this should create a picture from the camera and store it as mypicture.jpg
2. raspivid -t 5000 -o myvideo.h264
  1. this should create a video from the camera and store it as myvideo.h264
3. copy the below code to pycamtest.py and run it with python pycamtest.py. If everything is correctly set up you should see the output of your camera in a window.

import cv2
cv2.namedWindow("TestCV2")
vc = cv2.VideoCapture(0)
if vc.isOpened():
    rval,frame = vc.read()
else:
    rval = False
while rval:
    frame = cv2.flip(frame,-1)
    cv2.imshow("TestCV2", frame)
    rval, frame = vc.read()
    key = cv2.waitKey(20)
    if key ==27:
        break
vc.release()
cv2.destroyWindow("TestCV2")

3. Set up twitter functionality

Set up a twitter account
Activate a twitter developer account in https://developer.twitter.com/ and generate your API keys which you will need in the next step
Obtain your API keys and copy them to /home/pi/dev/birdfeederAI/auth.py
1. cat>/home/pi/dev/birdfeederAI/auth.py
2. consumer_key = 'puthereyourconsumerkey'
3. consumer_secret = 'puthereyourconsumersecret'
4. access_token = 'puthereyouraccesstoken'
5. access_token_secret = 'puthereyouraccesstokensecret'
6. Ctrl-C

4. Code your birdfeeder

Code your program, you can use my code to set up a headless tweeting bird detecting camera 🙂
1. git clone https://github.com/Rogeman/birdfeederAI.git

import numpy as np
import cv2
import random
import os
import logging
from twython import Twython
from twython import TwythonError

from auth import (
        consumer_key,
        consumer_secret,
        access_token,
        access_token_secret
)
twitter = Twython(
        consumer_key,
        consumer_secret,
        access_token,
        access_token_secret
)


confidence_thr = 0.5
CLASSES = ["background", "aeroplane", "bicycle", "bird", "boat",
    "bottle", "bus", "car", "cat", "chair", "cow", "diningtable",
    "dog", "horse", "motorbike", "person", "pottedplant", "sheep",
    "sofa", "train", "tvmonitor"]
COLORS = np.random.uniform(0, 255, size=(len(CLASSES), 3))
birdfeeder_dir=os.path.dirname(os.path.abspath(__file__))
logging.basicConfig(filename=birdfeeder_dir+'/log/birdfeederAI.log', level=logging.DEBUG, format='%(asctime)s %(message)s')
mobilenet_dir=birdfeeder_dir+'/MobileNet-SSD/'
net = cv2.dnn.readNetFromCaffe(mobilenet_dir+ 'deploy.prototxt' , mobilenet_dir+ 'mobilenet_iter_73000.caffemodel')
blob=None

def applySSD(image):
    global blob
    mybird = bool(False)
    blob = cv2.dnn.blobFromImage(cv2.resize(image, (300, 300)), 0.007843, (300, 300), 127.5)

    # pass the blob through the network and obtain the detections and
    # predictions
    net.setInput(blob)
    detections = net.forward()

    # loop over the detections
    for i in np.arange(0, detections.shape[2]):
        # extract the confidence (i.e., probability) associated with the
        # prediction
        confidence = detections[0, 0, i, 2]

        if confidence > confidence_thr:
            idx = int(detections[0,0,i,1])
            if CLASSES[idx]=="bird":
                mybird=bool(True)
    return mybird

def birdRatio(videoName):
    totalBirdFrames = 1
    totalFrames = 1
    vc2 = cv2.VideoCapture(videoName)
    if vc2.isOpened():
        rval2,frame2 = vc2.read()
    else:
        rval2 = False

    while rval2:
        birdinFrame = applySSD(frame2)
        rval2, frame2 = vc2.read()
        if (birdinFrame):
            totalBirdFrames = totalBirdFrames + 1
        totalFrames = totalFrames + 1

    vc2.release()
    return totalBirdFrames/totalFrames

videoLength=8*60*60*1000
randomsec=random.randint(0,videoLength)


#vc = cv2.VideoCapture(birdfeeder_dir+"/birds_video.mp4")
# If you want to record birds using your camera comment the above line and uncomment the below line. If you want to find birds in a video uncomment the line above and comment the line below 🙂
vc = cv2.VideoCapture(0)
vc.set(cv2.CAP_PROP_POS_MSEC, randomsec)
if vc.isOpened():
    width = vc.get(cv2.CAP_PROP_FRAME_WIDTH)
    height = vc.get(cv2.CAP_PROP_FRAME_HEIGHT)
    fps = vc.get(cv2.CAP_PROP_FPS)
    fcount = vc.get(cv2.CAP_PROP_FRAME_COUNT)
else:
    logging.error('Can\'t open video')

    exit()

recording= False
framerecorded = 0
framecounter = 0
birdinFrame=False
fourcc = cv2.VideoWriter_fourcc(*'h264')
#out = cv2.VideoWriter('output.mp4',fourcc,20.0,(640,480))
out = cv2.VideoWriter(birdfeeder_dir+'/output.mp4',fourcc,fps,(int(width),int(height)))

if vc.isOpened(): # try to get the first frame
    rval, frame = vc.read()
    (h, w) = frame.shape[0] , frame.shape[1]
else:
    rval = False

logging.debug('Started main loop')
while rval:
    #You enter this loop once per frame
    rval, frame = vc.read()
    #uncomment the below line if you need to flip the camera upside down.
    frame = cv2.flip(frame,-1)
    key = cv2.waitKey(20)
    if key == 27: # exit on ESC
        break
    framecounter = framecounter + 1
    if (framecounter > 60):
    # Write frame to disk every 60 frames so we can see what the camera is seeing
        framecounter = 0
        cv2.imwrite(birdfeeder_dir+"/webserver/currentframe.jpg",frame)
    if (birdinFrame==False):
        #Check if this frame has a bird in it
        birdinFrame= applySSD(frame)
    if (birdinFrame== True and recording== False):
        #You have detected the first bird in a frame, start recording
        logging.info('Started recording video')
        recording=True
    if (recording == True):
        #write the frame to file keep track of how many frames you have saved.
        framerecorded = framerecorded + 1
        out.write(frame)
    if (framerecorded > 200):
        #after 200 frames stop recording
        logging.info('Checking recorded video')
        recording = False
        birdinFrame=False
        framerecorded = 0
        out.release()
        filename = birdfeeder_dir+"/output.mp4"
        birdsinvideo= birdRatio(filename)
        logging.debug('percentage of bird in video: '+birdsinvideo)
        if (birdsinvideo> 0.50):
            # if the recorded video has more than 50% of frames with a bird in it then tweet it
            logging.info('Tweeting bird video')
            video = open(filename,'rb')
            try:
                response = twitter.upload_video(media=video, media_type='video/mp4', media_category='tweet_video', check_progress=True)
                twitter.update_status(status='birdfeeder 0.5', media_ids=[response['media_id']])
            except TwythonError as e:
                logging.error('Twitter error:'+str(e))
            birdsinvideo=0
            video.close()
        randomsec=random.randint(0,videoLength)
        vc.set(cv2.CAP_PROP_POS_MSEC, randomsec)
        os.remove(birdfeeder_dir+'/output.mp4')
        out = cv2.VideoWriter(birdfeeder_dir+'/output.mp4',fourcc,fps,(int(width),int(height)))



vc.release()

5. Create an nginx webserver to see what the camera is seeing

Install Docker

sudo apt-get update && sudo apt-get upgrade
curl -fsSL https://get.docker.com -o get-docker.sh
sudo sh get-docker.sh
sudo usermod -aG docker pi
sudo reboot

Run nginx docker container serving documents from the birdfeeder webserver folder
1. docker run -it --rm -d -p 8080:80 --name web -v /home/pi/dev/birdfeederAI/webserver/:/usr/share/nginx/html nginx
Create an index.html which refreshes every 2 seconds showing currentframe.jpg

cat > /home/pi/dev/birdfeederAI/webserver/index.html
<html>
    <head>
        <title>Birdfeeder</title>
        <meta http-equiv="refresh" content="2" />
    </head>
    <body>
    <img src=./currentframe.jpg>
    </body>
</html>

Ctrl+C

You can now open a web browser to your raspberry pi’s ip address port 8080 and see what your camera is seeing

6. Add birdfeeder service to systemd

We add birdfeeder to systemd so it starts on boot.

Create a bash script that runs birdfeeder in a loop. Run it with nice so it does not consume 100% of cpu (running at 100% for long makes the sd card non-responsive and the system unstable).
1. vim /home/pi/dev/birdfeederAI/bin/birdfeeder.sh
2. chmod +x /home/pi/dev/birdfeederAI/bin/birdfeeder.sh

#!/bin/bash
docker run -it --rm -d -p 8080:80 --name web -v /home/pi/dev/birdfeederAI/webserver/:/usr/share/nginx/html nginx
source /home/pi/dev/birdfeederAI/venv/bin/activate
while [ 1 -eq 1 ]
do
nice python /home/pi/dev/birdfeederAI/birdfeeder.py
done

Create service file
1. sudo vim /lib/systemd/system/birdfeeder.service

 [Unit]
 Description=birdfeeder service
 After=multi-user.target

 [Service]
 Type=idle
 ExecStart=/home/pi/dev/birdfeederAI/bin/birdfeeder.sh

 [Install]
 WantedBy=multi-user.target

Grant pemissions, add the service and reboot system
1. sudo chmod 644 /lib/systemd/system/birdfeeder.service
2. sudo systemctl daemon-reload
3. sudo systemctl enable birdfeeder.service
4. sudo reboot

One thought on “Artificial intelligence tweeting Bird Feeder”

Pingback: Caught my first bird in the wild! | My Map of things

Enter your comment here...

Fill in your details below or click an icon to log in:

Email (required) (Address never made public)

Name (required)

Website

You are commenting using your WordPress.com account. ( Log Out / Change )

You are commenting using your Twitter account. ( Log Out / Change )

You are commenting using your Facebook account. ( Log Out / Change )

Cancel

Connecting to %s

My Map of things

So much to do, so little time!!