Available online at www.sciencedirect.

com

ScienceDirect
IFAC PapersOnLine 55-32 (2022) 48–53
A Vision-based Navigation System for an Agricultural Autonomous Tractor
A
A Vision-based
Vision-based Navigation
Navigation SystemSystem forfor an
an Agricultural
Agricultural Autonomous
Autonomous Tractor
Tractor
A Vision-basedSristi
Navigation System for an Agricultural Autonomous
Saha*, Tsuyoshi Morita*, Ricardo Ospina**, Noboru Noguchi** Tractor
Sristi Saha*,
Sristi Saha*, Tsuyoshi Morita*, Morita*, Ricardo Ospina**, Noboru Noboru Noguchi**
Sristi Saha*, TsuyoshiTsuyoshi Morita*, Ricardo Ricardo

Ospina**,
Ospina**, Noboru Noguchi** Noguchi**
Sristi Saha*, *Graduate
Tsuyoshi School of Agriculture,
Morita*, Ricardo
 Hokkaido
Ospina**, University,
Noboru Noguchi**
*Graduate
*Graduate School
School of
of Agriculture,
 Hokkaido University,
Sapporo,
*Graduate
Sapporo,
Japan
School
Japan of Agriculture,
(email:
(email: Agriculture, Hokkaido
Hokkaido University,
saha96@bpe.agr.hokudai.ac.jp)
saha96@bpe.agr.hokudai.ac.jp) University,
Sapporo,
*Graduate
**Research
Sapporo, Japan
School
Faculty
Japan (email:
(email: saha96@bpe.agr.hokudai.ac.jp)
ofofAgriculture,
Agriculture, Hokkaido
HokkaidoUniversity,
saha96@bpe.agr.hokudai.ac.jp) University,
**Research
**Research
Sapporo, Japan
Sapporo, Faculty
Faculty
Japan of Agriculture,
of
(email:
(email: Agriculture, Hokkaido University,
Hokkaido
saha96@bpe.agr.hokudai.ac.jp)
noguchi@cen.agr.hokudai.ac.jp) University,
**Research Faculty of Agriculture, Hokkaido University,
Sapporo, Japan
Sapporo,
**Research Japan
Faculty(email:
(email:of noguchi@cen.agr.hokudai.ac.jp)
noguchi@cen.agr.hokudai.ac.jp)
Agriculture, Hokkaido University,
Sapporo, Japan (email: noguchi@cen.agr.hokudai.ac.jp)
Sapporo, Japan (email: noguchi@cen.agr.hokudai.ac.jp)
Abstract: RTK-GNSS (Real-time Kinematic-Global Navigation Satellite System) signal is necessary for
Abstract: RTK-GNSS
Abstract: RTK-GNSS (Real-time (Real-time Kinematic-Global
Kinematic-Global Navigation Navigation Satellite
Satellite System)
System) signal
signal is is necessary
necessary for for
the safe navigation
Abstract: RTK-GNSS of (Real-time
an autonomous tractor. However,
Kinematic-Global the main
Navigation problem
Satellite associated
System) signalwith this typefor
is necessary of
the
the safe navigation
safe
Abstract: navigation
RTK-GNSS of (Real-time
of an autonomous
an autonomous tractor. However,
tractor.
Kinematic-Global However, the main
the
Navigation main problem
problem
Satellite associated
associated
System) signal with
with
is this type
this
necessary typefor of
of
precise
the safe GNSS
navigationis thatofthe an RTK correction
autonomous signalHowever,
tractor. might not the be available
main at all geographical
problem associated locations.
with this type Treeof
precise
precise
canopies GNSS
the safeGNSS navigation
present is
is that
that
close the
ofthe
to RTK
anRTK correction
correction
autonomous
the tractors’ signal
signal
tractor.
vicinity might
might
However,
obstruct not
not be
be
the
satellite available
available
main
signals at
at
problem
and all
all geographical
geographical
associated
degrade locations.
locations.
with
the position this type
accuracy.Tree
Treeof
precise
canopies GNSS
present is that
close the RTK
to RTK correction
the tractors’
tractors’ signal
vicinity might
obstruct not be
satellite available
signals at
and all geographical
degrade the locations.
position accuracy.Tree
canopies
precise
This
canopiescan present
GNSSlead
present is
the close
that to
the the
autonomous correction
tractor vicinity
to signalobstruct
travel might
outside satellite
not ofbeits signals
available and
at
predetermined degrade
all the
geographical
path, position
which accuracy.
locations.
is unsafe. Tree
To
This
This
avoidcan
canopiescan
such lead
lead
present theclose
the close
situations,
to the tractors’
autonomous
autonomous
to thestudy
this tractor
tractor
tractors’
vicinity
to develop
to
aimsvicinity
to
obstruct
travel
travel outside
outside
obstruct
satellite
of its signals
of its
satellite
a vision-based
and degrade
predetermined
predetermined
signals
road and path,
path,
degrade
detection
the position
which
which
the
system, is
position
which
accuracy.
is unsafe.
unsafe.
accuracy.
does not To
To
This
avoid can
such lead the
situations, autonomous
this study tractor
aims to
to travel
develop outside
a of its
vision-based predetermined
road detection path, which
system, is
which unsafe.
does To
not
avoid
This
rely on
avoid such
can leadsituations,
RTK-GNSS.
such situations, this
this study
the autonomous
This system
study aims
aimsto
tractor
aims totodevelop
to travel
develop aa vision-based
detectoutside
both of its and
paved
vision-based road
predetermined
unpaved
road detection
roads
detection system,
path,
system, ruralwhich
inwhich does
areas.not
is unsafe.
farm
which does To
It
not
rely
rely onsuch
RTK-GNSS. This systemaims aimstoto to detect
detect both both paved and androad unpaved roadssystem,
in rural
ruralwhich
farm doesareas.not It
takeson
avoid
rely ontheRTK-GNSS.
RGB
RTK-GNSS. imagesThis
situations, this
This system
study
obtained
systemfrom aims
aimsantodevelop
HD onboard
detect both paved
a vision-based
cameraand
paved andunpaved
segments
unpaved roads
detection in
the road
roads in rural farm
surface
farm and areas.
areas. roadIt
It
takeson
takes
rely theRTK-GNSS.
the RGB images
RGB imagesThis obtained
obtained
system from
from
aims antoHD
an HD onboard
onboard
detect both cameraand
camera
paved andunpaved
and segments
segments the road
the
roads road
in surface
surface
rural farm and
and
areas. road
roadIta
edges.the
takes TheRGB segmentation
images obtainedis performed from an by HD usingonboard
several cameraimage filters, the sliding
and segments the window
road surfacemethod, andand road
edges.the
edges. TheRGB
The segmentation
segmentation is performed
is performed by HD
by usingonboard
using several camera
several image filters,
image filters, the sliding
the sliding window
window method,
method, and
and a
takes
set of rules
edges.
set of
images
The segmentation
of rules
rules that are
obtained
that are determined is performed
are determined
determined
from
manually.
manually.
an
by using Finally,
several
Finally,
the image
the image
and
lateralfilters,
lateralfilters,
segments
error obtained
the sliding
error obtained
obtained
the
from road
window
from
surface
the machine
the machine
machine
and
method, vision and aa
road
vision
set
edges.
system
set The that
segmentation
is transmitted
of rules is performed
to the tractor’s
that are determined manually.by
automatic
manually. Finally,
using several the
navigation
Finally, lateral
the lateralsystem.error the
errorExperimentalsliding
obtained from from
window
runs the method,
theperformed
machine vision vision
at thea
and
system
system
set of is transmitted
is
rules transmitted
that are to the
to the tractor’s
determined tractor’s automatic
automatic
manually. navigation
navigation
Finally, the system.
system.
lateral error Experimental
Experimental
obtained from runs
runs theperformed
performed
machine at the
at the
vision
experimental
system farm of Hokkaido
is transmitted to the tractor’s University
automaticshowed that the system.
navigation lateral error calculatedruns
Experimental by this system is
performed at less
the
experimental
experimental
system
than 0.2ismtransmitted
forfarm
farm of
of
unpaved Hokkaido
Hokkaido
to the
roads University
University
tractor’s
and 0.4automatic
m forshowed
showed that
that
navigation
paved the
the
roads. lateral
lateral
system.
This shows error
error calculated
calculated
Experimental
that the system by
by
runs this
this system
system
performed
can is
detectis less
less
atboth
the
experimental
than 0.2 munpaved
forfarm of
unpaved Hokkaido
roads andUniversity
0.4 m showed
for paved that the
roads. lateral
This error
shows calculated
that the by
system this
cansystem
detect is less
both
than
paved
than 0.20.2
experimental
and m forfarmunpaved
m for unpaved of
roads. roads
Hokkaido
Future
roads and
and 0.4
University
work m for
addresses
0.4addresses paved
showed
m for paved roads.
that the
considerations This
lateralto
roads. Thistoshows shows
error
make that the
calculated
the system
thatsystem system
the systemby
more can
this detect
system
robust.
can is
detect bothboth
less
paved
paved
than 0.2 and
andmunpaved
unpaved
for unpaved roads. Future
roads.roads
Future and work
work
0.4 addresses
mopen considerations
considerations
for access
paved roads.under
Thisto make
tothe
make
shows the
the
that system
the systemmore
more robust.
robust.
can detect both
Copyright
paved and © 2022 The
unpaved Authors.
roads. Future Thiswork is an
addresses article
considerations makeCC theBY-NC-ND
system license
more robust.
Keywords:
paved and Image
unpaved
(https://creativecommons.org segmentation,
roads. Future sliding
work window
addresses
/licenses/by-nc-nd/4.0/) algorithm,
considerations lateral
to error,
make therobot
systemnavigation,
more robot
robust. vision,
Keywords:
Keywords: Image
Image segmentation,
segmentation, sliding
sliding window
window algorithm,
algorithm, lateral lateral error,
error, robot
robot navigation,
navigation, robot robot vision,
vision,
Keywords:
road Image
edge detection, segmentation, sliding
precision agriculture. window algorithm, lateral error, robot navigation, robot vision,
road
road edge
edge
Keywords: detection,
detection,
Image precision
precision
segmentation, agriculture.
agriculture.
sliding window algorithm, lateral error, robot navigation, robot vision,
road edge detection, precision agriculture.
road edge detection, precision agriculture. 
 buildings and trees are present nearby, the RTK fixed
 buildings and trees
1. INTRODUCTION  buildings
solution
buildings and
drops
and to trees
trees theare floatpresent
are
are present
solution,
present
nearby,
nearby,
producing
nearby,
the
the
the RTK
RTK
some fixed
RTK fixed
data
fixed
1.
1. INTRODUCTION
INTRODUCTION solution
solution
buildings
jumps as drops
drops
and
well to
to
trees the
the
(Morales float
float
are solution,
solution,
present
and producing
producing
nearby,
Tsubochi, the
2007). some
some
RTK
As 70% data
data
fixed
of
1. INTRODUCTION solution
jumps as drops
well to the float
(Morales and solution,
Tsubochi, producing
2007). Assome70% data
of
According to the Agriculture1. INTRODUCTION and Forestry census of Japan solution jumps
jumps as
Japan’s asland well
drops
wellarea(Morales
to istheoccupied
(Morales and
float Tsubochi,
andsolution,
by mountains
Tsubochi, 2007).
producing
2007). andAs Assome70%
hills,
70% of
data
large
of
According to the Agriculture and Forestry census of Japan Japan’s
Japan’s land
land area
area is
is occupied
occupied by
by mountains
mountains and
and hills,
hills, large
large
According
(2020),
According to
the to the Agriculture
number
the Agriculture and Forestry
of core agricultural
and Forestryworkerscensus
census (those
of who jumps
of Japan
Japan tree
Japan’s
tree
asland
canopies
canopies
wellarea(Morales
are
are
present
is and
occupied
present
in most
in most
Tsubochi,
of the2007).
by mountains
most of the rural
andAs
rural
rural
farm 70%
hills,
farm
of
areas.
large
areas.
(2020),
(2020), the to
the number ofincore
core agricultural workers (those who
has Japan’s tree
Thus,canopies
inland are
thearea present
is occupied in of
by mountains the andthefarm
hills, areas.
large
According
are the number
usually
(2020), engaged
number of
the Agriculture
of core agricultural
and Forestry
agriculture
agricultural workers
for work)
workerscensusin(those
Japan
(those who
of Japan
who tree
Thus, canopies
in the arepresence
present
presence
of
in most
of
tree
tree
canopies,
of the rural
canopies, thefarm drop
drop
in
areas.
in
are
are usually
usually
(2020), engaged
engaged
thebynumber
22.4% of in
in agriculture
agriculture for
for work)
work) in
in Japan
Japan has
has Thus,
tree in
canopies the arepresence
present of
in tree
most canopies,
of the rural thefarm drop in
areas.
decreased
are usually engaged incore
compared agricultural
to 5 years
agriculture workers
for work) in(those
ago, amountingJapan who to Thus,
has measurement in the accuracy
presence is unavoidable.
of tree The next
canopies, the problem
drop is
in
decreased
decreased
are usually by
by 22.4%
22.4%
engaged compared
compared
in to
to
agriculture 5
5 years
years
for work)ago,
ago, amounting
amounting
in Japan to
to
has measurement
measurement
Thus, in
related to the RTK the accuracy
accuracy
presence is
is unavoidable.
unavoidable.
of
usageis method. tree The
The
canopies,
A userThe next
next
calculatesthe problem
problem
drop
the rover is
is
in
only aboutby1.36
decreased 22.4% million
comparedpeople.to Among
5 years these
ago, 1.36 million
amounting to measurement accuracy unavoidable. next problem is
only
only about
about
decreased by1.36
1.36
22.4% million
million
comparedpeople.
people. to Among
Among
5 years these
these
ago, 1.36
1.36
amounting million
million to related
related
measurement
position to
to bythe
the RTK
RTK
accuracy
receiving usage
usage is
the method.
method.
unavoidable.
correction A
A user
user The
data calculates
calculates
next
from a the
the
problem rover
rover
reference is
core about
only agricultural
1.36 workers,
million 69.6%Among
people. are aged these 651.36 years and related to the RTK usage method. A user calculates the rover
million
core
core
only agricultural
agricultural
about 1.36 workers,
workers,
million 69.6%
69.6%
people. are
are
Among aged
agedthese 65
65 1.36years
years and
and
million position
position
related
station to by
bythe
located receiving
receiving
RTK at usage
a the
the
known correction
correction
method. A
position data
data
user from
from
calculates
(Network-based aa reference
reference
the rover
RTK)
above,agricultural
core which results in the average
workers, 69.6% age are of aged 67.865years years for and
the position by receiving the correction data from a reference
above, which results in the average
69.6%in age of 67.8 years for the station
station located
located at
at a known positionthe (Network-based RTK)
above,
core which
agricultural
agricultural
above, which results
working
results in
workers, the
population
in the average
average age
are of
aged
Japan.
age of 67.8
If 65
67.8 years
this years
years for
for the
and
the (Tsuji
situation position
station et located at aa known
byal.,receiving
2013). the
However,
known position
correction
position (Network-based
data from a reference
Network-based
(Network-based RTK)
RTK
RTK)
agricultural
agricultural
above, which working
working
results population
population
in the average in
in Japan.
Japan.
age of If
If
67.8 this
this
years situation
situation
for the (Tsuji
(Tsuji
station
method et
et al.,
al.,
located
requires 2013).
2013).
at aa fast However,
However,
known data positionthe
the Network-based
Network-based
(Network-based
communication link, RTK
RTK
RTK)
which
continues,
agricultural Japan’s
working food self-sufficiency
population in Japan. rate will
If thisdrop further,
situation (Tsuji et al., 2013). However, the Network-based RTK
continues, Japan’s food self-sufficiency rate will drop further, method
method
(Tsuji requires
requires a
a fast
fast data
data communication
communication link,
link, which
which
continues,
agricultural
and the future
continues, Japan’s
working food
of Japanese
Japan’s self-sufficiency
foodpopulation
agriculture
self-sufficiencyin Japan. rate
itself
ratewillwill
If this
will drop
be
drop further,
atsituation
risk. To requires
further, method etrequires al., internet
good 2013).
a fastHowever,
connectivity. theRural
data communication Network-based
areas,link, which RTK
are
which
and thethis,
future ofnecessary
Japanese agriculture itself will be at risk.
risk. To method requires good by internet connectivity. Rural areas, which are
and
and the
solve the future
continues, it isof
of Japanese
Japan’s
future agriculture
food self-sufficiency
Japanese to start a new
agriculture itself
rate
itself will
typewillof
will be
drop
be at To requires
To
further,
agriculture
at risk. characterized good
good internet
requires requires internet fastconnectivity.
asmall data communication
populations
connectivity. Rural
and areas,
Rural smalllink,
areas, which are
which
industrial
which are
solve
solve
and thethis,
this, it
it
future is
is
of necessary
necessary
Japanese to
to start
start
agriculture a
a new
new
itself type
type
will of
of
be agriculture
agriculture
at risk. To characterized
characterized
requires good
activities, havebylimitedby
by small
small
internet populations
populations
connectivity.
orpopulations
no internet and and
and
Rural small
small
areas,
connectivity industrial
industrial
which
in manyare
that
solve realizes
this, it islabour-saving
necessary to start and ahigh-quality
new type of production. agriculture characterized small small industrial
that realizes
that
solve realizes labour-saving
this, ittheislabour-saving
Therefore,
that realizes necessary to start
development
labour-saving
andautomated
and
of
and
high-quality
ahigh-quality
new typeorofautonomous
high-quality agriculture activities,
production.
production.
production.
activities,
characterized
locations. Thus,
activities,
have
have limited
havebylimited
thesmall
limitedusage or
or no
no
of internet
internet
orpopulations
internet and
noRTK-GNSS
connectivity
connectivity
in small
rural farm
connectivity
in
in many
in many
industrial
areas
many
Therefore,
Therefore, the
the development
development of
of automated
andautomated orof autonomous
autonomous locations.
locations.
activities, Thus,
Thus, the
the
have depending
limitedusage
usage of
of
oronnotheRTK-GNSS
RTK-GNSS
internet in
in rural
rural
connectivity farm
farm areas
areas
in many
that realizes
agricultural
Therefore, the labour-saving
equipment
development is considered
of automatedto be or
high-quality or production.
the utmost can
autonomous be difficult
locations. Thus, the usage of above-mentioned
RTK-GNSS in rural factors.
farm areas
agricultural
agricultural
Therefore, equipment
equipment
the development is
is considered
considered
of automatedto
to be
be orof
of the
the utmost
utmost
autonomous can
can be
be
locations. difficult
difficult
Thus, depending
depending
the usage on
on
of the
the above-mentioned
above-mentioned
RTK-GNSS in rural factors.
factors.
farm areas
commercial and
agricultural societal is
equipment importance.
considered to be of the utmost can be difficult depending on the above-mentioned factors.
commercial
commercial
agricultural and
and societal
societal
equipment importance.
importance.
is considered to be of the utmost can
For be
the difficult
navigation depending
of an on the above-mentioned
autonomous tractor on factors.
the road,
commercial and societal importance. For thethe navigation
navigation of anan autonomous tractor on the
the road,
road,
commercial
Several autonomousand societal tractors that can be operated in open sky For
importance. precise
For the data of
of an autonomous
transmission
navigation without anytractor
autonomous on
obstructions
tractor on the is an
road,
Several
Several autonomous
autonomous tractors
tractors that
that can
can be
be operated
operated in
in open
open sky
sky precise
precise
For data
data
the data
essential transmission
transmission
navigation
requirement, of which without
without
an autonomous
if not any
any
met, obstructions
obstructions
tractor
can leadon the is
is
thetractoran
an
road,
fields
Several using an
autonomous RTK-GNSStractorsand and an
thatan inertial
caninertial
be operatedmeasurement
in open unitunit precise transmission without
sky essential requirement, which if not met, can lead the tractor any obstructions is an
fields
fields using an RTK-GNSS measurement essential
precise requirement,
data transmission which if
withoutnot met,
any can lead
obstructions the tractor
is an
fields using
Several
(IMU) as
using an
an RTK-GNSS
autonomous
navigation
RTK-GNSS sensorsand
tractors that
and an
havecaninertial
an be operated
been
inertial measurement
developed
measurement (Kiseunit
in open sky
et out
unit of its requirement,
essential
out ofoftimes
predetermined
its requirement,
predetermined
path,ifwhich
which
path,is
not met,
which
is unsafe.
can lead
is unsafe.
unsafe.
Therefore,
the tractor
Therefore,
at
at
(IMU) as navigation
navigation sensors have been etdeveloped
developed (Kise et essential
(IMU)
fields
al.,
(IMU) 2001,as
using an RTK-GNSS
Noguchi
as navigation etsensors and
al., 2001,
sensors have
have been
anTakai
inertial al., 2014).(Kise
measurement
been developed RTK-
(Kise et out
et
unit these
out
these of its
its
times
predetermined
a backup method
predetermined
a backup
path,
which
path,
method
ifwhich
is
not met,
which is
necessary,
is
necessary,
can
which
unsafe.
which
leadTherefore,
theprevent
can
Therefore,
can prevent
at
tractor
at
al.,
al.,
(IMU) 2001,
2001,as Noguchi
Noguchi
navigation et
et al.,
al.,
sensors 2001,
2001,
have Takai
Takai
been et
et al.,
al.,
developed 2014).
2014). RTK-
RTK-
(Kise et these
out of times
its
any dangerous a backup
predetermined method
situations path, is necessary,
which is which
unsafe. can prevent
Therefore, at
GNSS
al., can Noguchi
2001, provide centimetre-level
et al., 2001, Takai accuracy
et al., measurements
2014). RTK- these times a backup
any dangerous
dangerous methodin
situations inis
the absencewhich
necessary,
the absence
of RTK-GNSS.
can prevent
of RTK-GNSS.
RTK-GNSS.
GNSS
GNSS
al., can Noguchi
can
2001,
in real-time. provide
provide
But this centimetre-level
centimetre-level
et system
al., 2001, hasTakai
someaccuracy
accuracy measurements
al., measurements
etusability 2014). RTK- these
problems. any
Various times a backup
systems situations
method
based onin the absence
isdifferent
necessary, of
which
sensors can
haveprevent
been
GNSS can provide centimetre-level accuracy measurements any
Various dangerous
systems situations
based onin the absence
different of
sensors RTK-GNSS.
have well-been
in
in
GNSS
They real-time.
real-time.
can But
But
provide
are described this
this system
system
centimetre-level
below. RTKhashas
has some
some
offers usability
usability
accuracy
twousability problems.
solutions, Various systems based on different sensors have been
problems.
measurements
types of problems. Various
any systems
dangerous
developed for road based
situations
surface onin different
the
detection; absence
mostlysensors
of
for have
RTK-GNSS.
paved,
in real-time. But this system some developed for road surface detection; mostly for paved, been
well-
They are described below. RTK offers two types of solutions, developed for
They
float
They are
are described
in real-time.
and fix. But
describedRTKthisbelow.system
float
below. RTK
RTKhas
solution offers
some
needs
offers two attypes
leastof
twousability
types common Various
solutions,
of44problems.
solutions, marked,
developed for road
systems
well-maintained
marked, well-maintained
well-maintained
surface
road based
surfaceon detection;
different
roads.
detection;
roads.
mostly
Among
mostly
Among
sensorsfor paved,
these
for have
paved,
these
well-
been
sensors,
well-
sensors,
float
float
They and
and
are
satellites fix.
fix.
described
and RTK
RTK
offers float
float
below.
an solution
solution
RTK
accuracy needs
needs
offers
range two
of at
at least
least
types
about of 4
20cm common
common
solutions,
to 1m. marked,
developed
monocular
marked, for roadsensors
vision
well-maintained surfaceare roads.
detection;
one
roads. of Among
mostly
the
Among highly these
for paved,
used
these sensors,
well-
sensors
sensors,
float and and
satellites fix. offers
RTK an float solutionrange
accuracy needs of at
aboutleast20cm 4 common
to 1m. monocular vision sensors areroads.
one ofof the
the highly used sensors
satellites
float and
RTK fixed
satellites and
fix.
and offers
RTK an
solution
offers an accuracy
float
needs solution
accuracy range
at least needs
range of
5 common
of about
at
aboutleast 20cm to
4 common
satellites
20cm and monocular
to 1m.
1m.
marked,
as
monocular vision
data
vision cansensors
vision well-maintained provideare
sensors one
one ofAmong
areabundant highly
information
the highly used
these
used sensors
sensors,
about the
sensors
RTK
RTK fixed
fixed
satellites solution
solution
andsolution
offers anneeds
needs
accuracyat
at least
least
range 5
5 common
common
of satellites
satellites and
and as
as vision
vision
monocular data
data
vision can
can provide
provide
sensors are abundant
abundant
one of information
information
the highly used about
about the
the
sensors
offers
RTK fixedaccuracy within 5cm.
needs atBut,
leastwhen tallabout
5 common 20cmsuch
obstacles
satellites to 1m.as driving
and as vision scenes
data (Sivaraman
can provideand Trivedi,information
abundant 2013). Usingabout a vision
the
offers
offers
RTK accuracy
accuracy within
within 5cm.
5cm. But,
But, when
when tall
tall obstacles
obstacles such
such as
as driving
driving
as vision scenes
scenes
data (Sivaraman
(Sivaraman
can provide and
and Trivedi,
Trivedi,
abundant 2013).
2013).
information Using
Using a
a
about vision
vision
the
offers fixed
accuracysolution
within needs
5cm.atBut, leastwhen5 common
tall obstaclessatellitessuchandas driving scenes (Sivaraman and Trivedi, 2013). Using a vision
offers accuracy
2405-8963 Copyright within
© 2022 5cm.The But, when
Authors. Thistallis an
obstacles
open access sucharticle
as under driving
the CC scenes (Sivaraman
BY-NC-ND . Trivedi, 2013). Using a vision
licenseand
Peer review under responsibility of International Federation of Automatic Control.
10.1016/j.ifacol.2022.11.113
 Sristi Saha et al. / IFAC PapersOnLine 55-32 (2022) 48–53 49

sensor, many researchers have developed autonomous driving

systems that can track the lane markings on highways and
urban roads (Schneiderman and Nashman, 1992, Wang et al.,
2002, Chiu and Lin, 2005, Nguyen et al., 2017). However, a
tractor mostly travels on unstructured roads, such as rural
roads, which are less amenable to standard building (a) (b) (c)
techniques for a variety of reasons. There may be no lane
Fig.2. (a) 9x6 grid chessboard, (b) Input image with camera
markings and the edges may be spatially fuzzy. The overall
distortion, (c) Output image without camera distortion.
road shape may not follow smooth curves and the appearance
of the road itself can change drastically since mud, clay, sand,
gravel, and asphalt may all be encountered (Hu et al., 2004). 2.2 Image filtering
Therefore, this research proposes to create a vision-based
road detection system that can detect both paved and unpaved A combination of several image filters is used to filter the
roads, regardless of their type (structured or unstructured). undistorted images. According to the weather conditions,
selected filters are combined and applied to the image frames.
2. DEVELOPMENT OF THE MACHINE VISION These combinations are pre-defined and can be chosen
SYSTEM according to the weather conditions (as perceived by the
human eyes). The weather conditions are defined as
The system architecture is outlined in Fig. 1. The system ‘daytime-bright’, ‘daytime-cloudy’ etc. The image filters
relies on the HD RGB image frames (1080p, 24 FPS) used are described below.
obtained from a camera (KODAK, PixPro) onboard the
tractor. These image frames are then processed in a computer, RGB OpenCV’s ‘GaussianBlur’ Image with
which is equipped with an Intel Core i9-10980HK image Function applied reduced noise
(@2.40GHz) processor, 16 GB RAM, and Windows 10 Pro OpenCV’s colour
(64 bit) operating system. The computer has an NVIDIA filters applied
GeForce RTX 2070 Max-Q graphics card with 8 GB GDDR6 HSV HSL
RAM. The road detection algorithm is run on this computer image image
using Python 3.7.0 in the Anaconda distribution. This OpenCV’s morphological OpenCV’s ‘Sobel’
transformation applied function applied
algorithm can be run on a computer without a graphics card
also, as the python program does not use any library that Combined and
requires the graphics processing unit. The whole process is thresholded
comprised of 7 main processes, which are described below.
All these processes are performed in real-time.
Final output
Camera HD RGB Road image with
both edges in
onboard image detection
white pixels
the tractor frames algorithm

Fig. 3. Image processing flowchart.

Tractors’ Output:
automatic Lateral At first, the OpenCV’s ‘GaussianBlur’ function is applied to
navigation system error smooth out the images. Then several filters are applied to this
image. The first filter comprises converting the RGB image
Fig.1. Overview of the road detection system. frames to HSV (Hue, Saturation, and Value) colour space.
The HSV image frames are then thresholded to isolate the
desired colour range (such as the green colour of vegetation
2.1 Capturing the image frames and removing distortion
that is mostly present at the edges of unpaved roads). Then
the resulting image frames are morphologically transformed
The main process starts by capturing the HD image frames by
to further reduce noise. The next filter is comprised of
using the OpenCV (version 4.2.0) library’s ‘VideoCapture’
converting the RGB image frames to HSL (Hue, Saturation,
function. As all cameras introduce significant distortion to the
and Lightness) colour space. The HSL colour space can
images, it is necessary to remove the camera distortion. To do
detect lines and edges (e.g., white lane markings in paved
that, the camera intrinsic parameters are calculated by using a
roads) with proper thresholding. Then Sobel operator is
9x6 grid chessboard (Fig. 2a). The KODAK PixPro camera
applied on the lightness channel of HSL image frames to
has a fisheye lens, which causes the input image to look
emphasize the pixels. At last, selected filters are combined
curved (Fig. 2b). This curvature makes it difficult to obtain
with the help of python’s logical operators (and, or, not
real-world distances in the final steps of the process. To avoid
operator), and the final output image is obtained, which
that, the OpenCV’s ‘fisheye’ camera model is used to remove
shows the road surface in black and the surrounding in white
the distortion with the camera’s intrinsic parameters taken
in Fig. 3.
into consideration. In Fig. 2c, it is evident how the image is
stretched compared to the distorted image; both the road and
horizon lines do not look curved, they look straight.
50 Sristi Saha et al. / IFAC PapersOnLine 55-32 (2022) 48–53

2.3 Selecting the region of interest

In this step, the region of interest is selected manually by

fine-tuning. Usually, this depends on some external factors No edge
such as the field of view and the width of the road. Next, present on
the left side
OpenCV’s ‘warpPerspective’ function is applied to the
filtered images obtained from the previous step. This step
Fig. 5. Picture showing absence of one edge (unpaved road).
results in a bird’s eye view of the region of interest, which
facilitates the estimation of the road edges and the centre of
the road surface. The resulting images are shown in Fig. 4a. 2.5 Auto-tuning of threshold values used in image filtering

2.4 Road edge detection: sliding window algorithm As stated in section 2.3, weather conditions are defined as
‘daytime-bright’, ‘daytime-cloudy’, etc., and the set of
The next step consists of applying the sliding window threshold values used for each weather condition is constant.
algorithm to the warped image frames. In this algorithm, a But in reality, even if one of these weather conditions is
window of specified length Len, moves over the image frame prevalent, the light intensity still varies. So, constant
data, sample by sample. These windows are represented by threshold values do not work in real-time navigation. Thus,
the yellow rectangles shown in Fig. 4b. The statistic is adjustment of the threshold values is required during
computed over the data in each window. The output for each navigation. To do that, firstly, the width of the road is
input sample is the statistic over the window of the current calculated from the detected left and right edges, which are
sample and the length of the previous sample defined as Len- obtained from the sliding window algorithm. Then, it is
1. In the first-time step, to compute the first Len-1 outputs compared to the actual width of the road. If the calculated
when the window does not have enough data yet, the width is not the same as the actual width, the auto-adjustment
algorithm fills the window with zeros. In the subsequent time process starts. In this process, an iterator iterates over a range
steps, to fill the window, the algorithm uses samples from the of threshold values until the calculated width becomes the
previous data frame. same as the actual road width. This range is defined as the
current threshold value ± 20.

2.6 Displaying the final result

(a) (b) (c)

Fig. 4. (a) Warped image showing the region of interest, (b)
Sliding windows detecting the non-zero pixels, (c) Sliding
windows after stabilization.
(a) (b)
For this algorithm, the OpenCV, the NumPy, and the Pandas
libraries are used. There are 9 sliding windows along with the Fig. 6. Display of the final result: (a) unpaved (dirt) road, (b)
height (1080 pixels) of one image frame. Each window paved (asphalt) road.
searches for the non-zero pixels inside that window; note that
After getting the second-degree polynomials from the warped
the pixels in black are assigned zero value and one is
images, the coordinates in the polynomial are converted to
assigned to the pixels in white. Next, these non-zero pixel
real-world coordinates by inverting the warping. The same
values are stored in an array and a second-degree polynomial
OpenCV function “warpPerspective” is used to invert the
is fit on these pixel values. These second-degree polynomials
warping. Then, the final result is overlapped with the original
represent the left and right edges of the road and help in
input image; this is the image without any camera distortion.
determining the centre of the road. Fig. 4b shows the
In Fig. 6, the yellow polygon shows the road surface detected
detection result. While searching for the non-zero pixels, the
in the region of interest with respect to the left and right
sliding windows create a region of interest inside the frame.
edges. The red polyline shows the centre of the road.
If non-zero pixels are continuously found inside this region of
interest, the sliding window method gets stabilized (Fig. 4c).
2.7 Calculation of lateral error
As farm roads are mostly unstructured, sometimes one edge
can be missing (Fig. 5). For that scenario, the positions of the The lateral error is estimated from the tractor’s position
detected left and right edges are taken into account. A range relative to the road. To achieve this, the real-world distances
were manually calculated using a chessboard to relate the
of position values is also pre-defined for both edges. This
pixel distances with the real-world distances. At first, the
range depends on the width of the road. If the detected
horizontal and vertical lengths of the chessboard are
position of one of the edges does not fall inside the pre-
defined range, the centre is calculated from the other edge. In measured in meters. Then the same lengths are measured in
this case, the width of the road is halved and added (or pixel values from the image shown in Fig. 7a. Finally, these
subtracted) to (or from) the edge in consideration. two lengths are co-related to get the real-world distances
from the image frames.
 Sristi Saha et al. / IFAC PapersOnLine 55-32 (2022) 48–53 51

Actual centre centrelines are used as the reference data to assess the
of the frame efficiency of the vision-based system. Several experimental
(tractor’s position)
runs were performed on both types of roads. During the
Centre of road
obtained from experimental runs, the position data of the tractor is recorded
Lateral error
image processing using a Trimble alloy GNSS receiver. This RTK data is
recorded only for assessing the efficiency of the system. The
(a) (b) vision-based system did not use RTK-GNSS for the field
experiments. The results obtained from the field experiments
Fig. 7. (a) Manual calculation of real-world distances from are discussed in the next section.
chessboard, (b) Lateral error estimation.
Unpaved road (Hokkaido university experimental farm)
Fig. 7b represents the calculation of lateral error by using the 4769140

real-world distances obtained from the above-mentioned 4769120

method and the pixel values obtained from the image frames. 4769100

Northing
The left edge is shown by the blue area, the right edge is 4769080

shown by the orange area and the detected centre of the road 4769060

is represented by the red line. The green dashed line 4769040

represents the actual centre of the image frame. The distance 4769020
527440 527450 527460 527470 527480 527490 527500 527510 527520
between the green dashed line and the red line represents the Easting
lateral error in pixel values. It is then converted to real-world Left edge Right edge
distances and can be transmitted to the tractor’s automatic
navigation system pc using serial port communication as Fig. 8. Road edge data for an unpaved road inside Hokkaido
shown in Fig. 1. University, Japan.
Paved road (Hokkaido University poplar avenue)
3. EXPERIMENTAL PROCEDURE 4769580
4769570
4769560
The experiments were done in two ways. At first, manually 4769550
Northing

driven sample videos of different paved and unpaved roads 4769540

4769530
were recorded inside the Hokkaido University campus in 4769520
4769510
Japan. Then the vision-based algorithm was desktop-tested 4769500

inside the laboratory using these sample videos. This was 4769490
4769480
done to check if the algorithm can detect both paved and 527000 527020 527040 527060 527080 527100 527120 527140 527160
Easting
unpaved roads. After laboratory testing, it was confirmed that Left edge Right edge
the tractor can detect both paved and unpaved roads. So, field
experiments were performed after this in real-time to assess Fig. 9. Road edge data for a paved road inside Hokkaido
the efficiency of the developed algorithm. The experimental University, Japan.
setup and procedure are described below.

Before beginning the field experiments, road width and edge 4. EXPERIMENTAL RESULTS & DISCUSSION
data were manually obtained. For unpaved roads, the road
width is not constant. So, after measuring the width of Several field experiments were performed for both unpaved
different parts of the road, an average value was considered and paved roads under different weather conditions and
during the field experiments. Also, the road left and right during different times of the day. Some of the results for both
edges were recorded by using a Trimble R10 GNSS receiver. unpaved and paved will be discussed below.
The edges were determined by human judgement as there are
no defined edges. For paved roads, a constant width was 4.1 Unpaved road
taken into account and the same GNSS receiver was used to
record edge data. Field experiments were run on the unpaved road shown in
Fig. 8. As stated in section 3, the path travelled by the tractor
As seen in fig. 8 and 9, the left and right edge data are shown was recorded using RTK-GNSS and is shown by the grey-
in blue and orange dotted lines respectively. The unpaved coloured dotted line in Fig. 10. The approximate travel
road in Fig. 8 is about 110 m in length and an intersection distance was 112 m at a speed of 4.4 km/h. The weather was
divides the road into almost equal halves. As there are no mostly cloudy during the time of the experiment.
trees present immediately next to this road, the data is
continuous. The paved road in Fig. 9 is about 150 m long and The lateral error obtained from the machine vision system
surrounded by poplar trees on both sides. So, the RTK fixed during this experiment is shown in the red continuous line in
solution drops a lot here and thus discontinuous edge data is Fig. 11. From this graph, it can be seen that the lateral error is
obtained. mostly within 0.2 m. However, in the middle, a lateral error
value as big as 1 m can be seen. This is because of the
The edge data obtained from both the roads in consideration presence of an intersection in the middle of the road. As an
are used to calculate the true centrelines of the roads. These intersection does not have any edges, this part is travelled by
52 Sristi Saha et al. / IFAC PapersOnLine 55-32 (2022) 48–53

connecting the endpoints of the edges on both ends of the road, the condition regarding the width of the road was still
intersection. But the endpoints of the edges are not found true. So the tractor kept going to the left. But, at the extreme
inside the region of interest at all times. When this happens, end of this path, it can be seen that the tractor started coming
the sliding windows start searching for any non-zero pixel back towards the right, close to the actual centre. This
inside the region of interest, which if found, gets considered happened because the stabilization conditions were not met,
as edges and thus yields big values of lateral errors and the sliding windows started searching for the non-zero
sometimes. In case of the absence of non-zero pixels inside pixels again.
the region of interest, the previous data is considered to
calculate the edges. 4.2 Paved road

4769140
Path travelled by the tractor on the unpaved dirt road Field experiments similar to that of the unpaved road were
4769120
run on the paved road also. The grey-coloured dotted line in
Fig. 12 shows the path travelled by the tractor as recorded by
4769100
the RTK-GNSS. But, due to the presence of tree canopies on
Northing

4769080
both sides of the road, the RTK fixed solution dropped to the
4769060
float solution a lot. Thus, the tractor’s position data in Fig. 12
4769040
is a mix of both RTK fix and float solution data.
4769020
527440 527450 527460 527470 527480 527490 527500 527510 527520
Easting
In this experiment, the road does not have any white (or
Right edge Left edge Tractor's position
yellow) lane markings. The edges of the pavements on both
Fig. 10. The path travelled by tractor during the field sides are detected as left and right edges (Fig. 6b shows the
experiment on 10/11/2021 (unpaved dirt road). same road). The tractor travelled an approximate distance of
131 m at a speed of 3.2 km/h. The weather was sunny with
Distance between the tractor's position and the actual centre line
lots of shadows (because of the presence of tree canopies) on
1
Actual distance (m)

the road surface. Also, as the experiment was done in early

0.5
winter (09/12/2021), lots of fallen leaves were present on the
0 road surface.
-0.5
Path travelled by the tractor on the paved asphalt road
-1
4769580
Lateral error obtained from the machine vision system 4769570

1 4769560
4769550
Lateral error (m)

Northing

0.5 4769540
4769530
0 4769520
4769510
-0.5 4769500
4769490
-1 4769480
527000 527020 527040 527060 527080 527100 527120 527140 527160

Fig. 11. Comparison of lateral error obtained from machine Easting

Left edge Right edge Tractor's position
vision system and position data (unpaved dirt road). (+)
and (-) values indicate the left and right sides of the Fig. 12. The path travelled by tractor during the field
referral line respectively. experiment on 09/12/2021 (paved asphalt road).
The blue continuous line in Fig. 11 shows the distance
calculated between the tractor’s position and the actual centre The red continuous line in Fig. 14 shows that lateral error is
line of the road. If we consider this line, it can be stated that within 0.4 m at all times, but it oscillated a lot. There are two
the tractor travelled a straight path almost all the time except reasons behind this. The first is the presence of lots of
at the middle and the end of the road. In the middle portion, shadows and fallen leaves on the road surface, which leads to
because of the presence of the intersection, the tractor went misdetection of the edges. The second reason is the large
around 0.6 m to the left of the actual centreline. At the last width of the road. The road is 6 m in width, which implies
part of the road, the tractor kept going away from the actual that a small difference in pixel values leads to a big lateral
centre line towards the left side of the road. But red error value in metres.
continuous line shows that the lateral error did not change
much during this part. This happened because of certain In Fig. 14, the blue continuous line shows the distance
properties of the sliding window algorithm and the constant between the tractor’s position and the actual centre line. It is
width of the road, which is used as a conditional parameter. seen that the tractor keeps to the left side of the actual centre
As stated in section 2.4, the sliding windows get stabilized line almost the whole time. This happened mostly because of
when non-zero pixels are continuously encountered within the misdetection that took place. Also, another reason is that
the region of interest created by the sliding windows. So, the the sliding windows were unable to detect the right edge at
sliding windows stabilization took place at the last part of the some parts of the road (as the right edge was covered with
road and thus the lateral error did not change much. Also, the fallen leaves at some parts of the road). So, the centre of the
road width entered in the system is an average calculated road was calculated from the left edge and thus caused the
manually. So, even if the width increased at the end of the
 Sristi Saha et al. / IFAC PapersOnLine 55-32 (2022) 48–53 53

tractor to gradually move towards the left side of the road. REFERENCES
But, as the values of the lateral error are not very big, the
tractor travelled almost a straight path without going much to Agriculture and Forestry Census, Ministry of Agriculture,
the sides. Forestry and Fisheries, Japan. (2020).
https://www.maff.go.jp/j/tokei/sihyo/data/08.html
Accessed May 19, 2021.
Chiu, K., and Lin, S. (2005). Lane detection using color-
based segmentation. In IEEE Proceedings Intelligent
Vehicles Symposium, 706-711.
https://doi.org/10.1109/IVS.2005.1505186
Fig. 13. Presence of shadows and fallen leaves on the road Hu, M., Yang, W., Ren, M., Yang, J. (2004). A Vision Based
surface. Road Detection Algorithm. In IEEE Conference on
Distance between the tractor's position and the actual centre line
Robotics, Automation and Mechatronics, 2004, 2, 846-
0.8 850.
0.6 https://doi.org/10.1109/RAMECH.2004.1438028
0.4
Kise, M., Noguchi, N., Ishii, K., Terao, H. (2001).
(m)

0.2
0
Development of the Agricultural Autonomous Tractor
-0.2 with an RTK-GPS and a Fog. IFAC Proceedings
-0.4 Volumes, 34(19), 99–104. https://doi.org/10.1016/S1474-
Lateral error obtained from MV system
0.6
6670(17)33120-8
Morales, Y. and Tsubouchi, T. (2007). DGPS, RTK-GPS and
Lateral error (m)

0.4
0.2 StarFire DGPS performance under tree shading
0
environments. 2007 IEEE International Conference on
-0.2
-0.4 Integration Technology, 519 - 524.
-0.6 DOI: 10.1109/ICITECHNOLOGY.2007.4290370.
Nguyen, V., Seo, C., Kim, H., Boo, K. (2017). A study on
Fig. 14. Comparison of lateral error obtained from machine detection method of vehicle based on lane detetction for a
vision system and position data (paved asphalt road). (+) driver assistance system using a camera on highway. In
and (-) values indicate the left and right sides of the 11th Asian Control Conference (ASCC), 424-429.
referral line respectively. https://doi.org/10.1109/ASCC.2017.8287207
Noguchi, N., Reid, J.F., Manager, Zhang, Q., Will, J.D., Ishii,
5. CONCLUSION K. (2001). Development of Robot Tractor Based on RTK-
GPS and Gyroscope. In 2001 ASAE Annual Meeting.
In this paper, a vision-based road detection system has been https://doi.org/10.13031/2013.7297
presented. This system makes use of the colour information Schneiderman, H., and Nashman, M. (1992). Visual
obtained from the camera and defines the edges of the road. processing for autonomous driving. In Proceedings IEEE
This leads to the calculation of the lateral error, which is then Workshop on Applications of Computer Vision, Palm
provided to the automatic navigation system of the tractor for Springs, CA, USA, 164 - 171.
autonomous driving. From the field experiments, it can be https://doi.org/10.1109/ACV.1992.240315
said that this system can detect both unpaved and paved roads Sivaraman, S., and Trivedi, M.M. (2013). Looking at
in real-time. The major factors affecting this detection are the Vehicles on the Road: A Survey of Vision-Based Vehicle
width of the road, the weather condition (illumination), and Detection, Tracking, and Behavior Analysis. IEEE
the condition of the road (presence of proper edges). Transactions on Intelligent Transportation Systems,
14(4), 1773-1795.
It is to be kept in mind that this method is developed to act as https://doi.org/10.1109/TITS.2013.2266661
a backup method for situations when the RTK correction Takai, R., Yang, L., Noguchi, N. (2014). Development of a
signal cannot be received. So, this method needs to be robust crawler-type robot tractor using RTK-GPS and IMU.
enough to achieve that objective. The current system does not Engineering in Agriculture, Environment and Food, 7(4),
produce precise results at all times. Many external factors 143–147. https://doi.org/10.1016/j.eaef.2014.08.004
(such as variable light conditions, the sudden growth of Tsuji, H., Miyagawa, K., Yamaguchi, K., Yahagi, T.,
vegetation in the middle of unpaved roads, shadows present Oshima, K., Yamao, H., Furuya, T. (2013).
on the road surface, reflection from the potholes, etc.) affects Modernization of GEONET from GPS to GNSS. Bulletin
this system to some extent. So, future work will address all of the Geospatial Information Authority of Japan, 61.
these issues and make the system more suitable for https://warp.da.ndl.go.jp/info:ndljp/pid/8440829/www.gsi
agricultural roads. Also, future work will include different .go.jp/common/000085715.pdf
types of unpaved roads (other than dirt roads), paved roads Accessed May 19, 2021.
with proper (white or yellow) lane markings, and curved Wang, R., Xu, Y., Libin, Zhao, Y. (2002). A vision-based
roads as well. road edge detection algorithm. In Intelligent Vehicle
Symposium, IEEE, 1, 141-147.
doi: 10.1109/IVS.2002.1187942.